2022-03-07T23:23:26.879 INFO:root:teuthology version: 1.1.0-eeaeee38 2022-03-07T23:23:26.881 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/flannel rook/master} --name yuriw-2022-03-07_22:06:10-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi --verbose --archive /home/teuthworker/archive/yuriw-2022-03-07_22:06:10-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi/6725394 -- /home/teuthworker/archive/yuriw-2022-03-07_22:06:10-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi/6725394/orig.config.yaml 2022-03-07T23:23:26.918 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-03-07T23:23:26.989 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-03-07_22:06:10-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi/6725394 branch: wip-yuri-testing-2022-03-07-0958 description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/flannel rook/master} email: ceph-qa@ceph.io first_in_suite: false job_id: '6725394' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2022-03-07_22:06:10-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri-testing-2022-03-07-0958 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 94cb3430b74fb099b8927e3e6ad50dfd969173f4 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 94cb3430b74fb099b8927e3e6ad50dfd969173f4 kernel: hwe: true kubeadm: pod_network: flannel version: '1.21' rook: rook_image: rook/ceph:master workunit: branch: wip-yuri-testing-2022-03-07-0958 sha1: 94cb3430b74fb099b8927e3e6ad50dfd969173f4 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.a - - host.b - client.b - - host.c - client.c sha1: 94cb3430b74fb099b8927e3e6ad50dfd969173f4 sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri-testing-2022-03-07-0958 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 94cb3430b74fb099b8927e3e6ad50dfd969173f4 targets: smithi078.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDoh03aO5u0hj92SBCpYuFyYiZT19FukfWsx6deWKVkvGmmWakMFtkMGFNMVG/BDKBjyPZvkTYKOSE0WFiUSMh6LfkUKHI5suDjBdgdlkdwkhEnoQdjyL8VKHnhIv1LvdxqMFVD2AbvbZrFhFwSgjmACFAteC+ZYKPloFwKHeQZrMKk6fH95kW3WjBa2dqCUBlfg6xH3HdD1OcgH/o4NP+SRr1BzddBqLxySnTjyq5A6TLCGA6praS+8NXBbCnCuBMwEQJptzeA0Ttuw0Si9Tfm8gaeJ1UUgURofoEkcmCjrB9koJenRV+BUXn7LMal6e20sKm1b/7J5KQEBcQrI8BmAyAckH3+WP+dhM8nLB++wRLs8O7ZEStDfA7xtK0VHaCrccLetb4G7mYh1+Jwkh1GN7YRHTQq0Bx2HByft8+X4+rVPvB6T8NYoh3J3QcpJWUYs/tPxhhv4FErJCkNdAtuPG5EkzIPvVNC3DwQobWei3dnG+eCuktUO/tO5JZZK10= smithi166.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoEVIZlpmZ70Vy4l7WCn9h+Y75FrwvfZZls6/bD0HCQ/5Xsjcmyoo+1aNZRiJBeEX5ftHM60x8lmxPDcKCmK+UwtWq15zsNDm/XBaxK7HvYjVgBgSq+EwD2iQgm4NW9rBuiK6ChmuhxMhuebCQPsAdXKhcq1hlx/pX3OQo3PqeWOT/V6X1pV0ZvBN/KyRHkm1EEh+iNLD2swMc4CpKUWkyn9XzMSt3QFZV1sDIx7ez1eLBGsYDSmSiD0XdmiWuSvi7vaxfrSlXDGsvVfSLi8t9DsKJTqImiAbnYvyqSsygAJY8xXJrGyjD3Djp8k3gcRlxT0yNQembjtEORjLwJwGGpFIaOnfbyN5Vi5UdhUe5pyFUSd+N+hzUPU2Q5VRrRp84VRlrnCH6vg2uAyL+wPGjcEJFZeXZqxjPQPz24l6NCHUmOwSj6vAsRUAR1AJ+RX0n2VeRd6wjdq8vRgpSz/doGnr3SQvj6h79ZVCKMgzGNqfdZsLM89CuFnA3NhyhsC8= smithi170.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOr2kJonQjNOoZkORs95E9voqPPsgQPSv5d8HQ/eorbLCjdZgQJ4Pm3Eaz5ZWTPwm92Z31NOTlxWdMTKHvHu8F7cIul8h9lb5YuCN1wxFrpfEQJHCxqeKphMLfNUjFBdIemOMOvHoDjiAPXLOATdu7lAl2o03LMT/U9flvOQ3E/6oVcHLOtPihyPg8fxZ2+hj1uNbk/4kA+2Xs7VZNGfvYoMfH6ROuI2Q9fTtMDakBZhegO068dx/26aCskSPiaUMzWp41DtefSUMkxO8z0Ja1iCUS4aFiutytBqrbvc1baYU1cHgcxm1vQA3I0HlFprpkLXDH2JR5CgyxFYtXa4i3Uj4OIyEyiWL3FAnmYe/XpZqAOeiZeHgl4m1JQKQ+O5xeGD36OQ2MvbcNO5WkW+Hu/bnY0wcS7y8WGbyUbjLrIQewQnn55GrrJQDUAqskcIfO3tlmUIJLQ0WueenLgdBA+fgSl7dn7kCSfroZ5TqAYGbmNKLTSQqB2rk+FU9fJQ0= tasks: - kubeadm: null - nvme_loop: null - rook: null - rook.shell: - ceph -s - ceph orch status - ceph orch ps - ceph orch ls - ceph orch device ls - exec: host.a: - "set -ex\ntoolbox() {\n kubectl -n rook-ceph exec -it deploy/rook-ceph-tools\ \ -- \"$@\"\n}\norig_num_osd=`toolbox ceph osd stat | cut -f3 -d \" \"`\ntoolbox\ \ ceph orch osd rm 0 --force\nremoved_pv=\"\"\nwhile [ \"$removed_pv\" = \"\"\ \ ]\ndo\n removed_pv=`kubectl get pv | grep Released | cut -f1 -d \" \"`\n\ \ sleep 3s\ndone\ntarget_path=`kubectl get pv $removed_pv -o jsonpath='{.spec.local.path}'`\n\ host=`echo $removed_pv | cut -f1 -d \"-\"`\ntoolbox ceph orch device zap $host\ \ $target_path --force\nzap_completion=\"0\"\nwhile [ \"$zap_completion\" =\ \ \"0\" ]\ndo\n zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap\ \ -o jsonpath='{.status.succeeded.path}'`\n sleep 3s\ndone\nkubectl patch\ \ pv $removed_pv -p '{\"spec\":{\"claimRef\": null}}'\ntoolbox ceph orch apply\ \ osd --all-available-devices\nkubectl delete job rook-ceph-device-zap -n rook-ceph\n\ num_osd=\"0\"\nwhile [ \"$num_osd\" != \"$orig_num_osd\" ]\ndo\n echo \"\ waiting for osd to come back up\"\n num_osd=`toolbox ceph osd stat | cut\ \ -f3 -d \" \"`\n sleep 30s\ndone\n" - rook.shell: commands: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch apply rgw foo - ceph orch apply mds foo - ceph orch apply rbd-mirror - ceph orch apply nfs foo --port 12777 teuthology_branch: master teuthology_sha1: eeaeee38264fad8a01fd1ce912d8908676ed64bc timestamp: 2022-03-07_22:06:10 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-03-07T23:23:26.989 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa; will attempt to use it 2022-03-07T23:23:26.990 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks 2022-03-07T23:23:26.991 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-03-07T23:23:26.993 INFO:teuthology.task.internal:Checking packages... 2022-03-07T23:23:27.018 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '94cb3430b74fb099b8927e3e6ad50dfd969173f4' 2022-03-07T23:23:27.019 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-03-07T23:23:27.020 INFO:teuthology.packaging:ref: None 2022-03-07T23:23:27.020 INFO:teuthology.packaging:tag: None 2022-03-07T23:23:27.020 INFO:teuthology.packaging:branch: wip-yuri-testing-2022-03-07-0958 2022-03-07T23:23:27.020 INFO:teuthology.packaging:sha1: 94cb3430b74fb099b8927e3e6ad50dfd969173f4 2022-03-07T23:23:27.022 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri-testing-2022-03-07-0958 2022-03-07T23:23:27.180 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-11002-g94cb3430-1focal 2022-03-07T23:23:27.181 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-03-07T23:23:27.189 INFO:teuthology.task.internal:no buildpackages task found 2022-03-07T23:23:27.190 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-03-07T23:23:27.237 INFO:teuthology.task.internal:Saving configuration 2022-03-07T23:23:27.250 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-03-07T23:23:27.272 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-07T23:23:27.296 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-07_22:06:10-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi/6725394', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-07 23:17:58.219983', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFabY/ULMTC/fsPrwRe+8vEtKVHi1gfE2gPxQ8m6cpw2'} 2022-03-07T23:23:27.320 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi166.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-07_22:06:10-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi/6725394', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-07 23:17:58.221127', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE0yEpoee9qllEz+7xRRKthnAZjSkucT5HhnzsFDU7tf'} 2022-03-07T23:23:27.347 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-03-07_22:06:10-rados-wip-yuri-testing-2022-03-07-0958-distro-default-smithi/6725394', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-03-07 23:17:58.218544', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEmBAkvA7vBgaFXVzcAd0WU+Jdh1Kw5bHIL7P/dyC57/'} 2022-03-07T23:23:27.348 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-03-07T23:23:27.356 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-03-07T23:23:27.356 INFO:teuthology.task.internal:roles: ubuntu@smithi166.front.sepia.ceph.com - ['host.b', 'client.b'] 2022-03-07T23:23:27.356 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['host.c', 'client.c'] 2022-03-07T23:23:27.357 INFO:teuthology.run_tasks:Running task console_log... 2022-03-07T23:23:27.484 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb210a0e9d8>, signals=[15]) 2022-03-07T23:23:27.486 INFO:teuthology.run_tasks:Running task internal.connect... 2022-03-07T23:23:27.503 INFO:teuthology.task.internal:Opening connections... 2022-03-07T23:23:27.503 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi078.front.sepia.ceph.com 2022-03-07T23:23:27.505 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T23:23:27.561 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi166.front.sepia.ceph.com 2022-03-07T23:23:27.562 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi166.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T23:23:27.614 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi170.front.sepia.ceph.com 2022-03-07T23:23:27.615 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T23:23:27.673 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-03-07T23:23:27.683 DEBUG:teuthology.orchestra.run.smithi078:> uname -m 2022-03-07T23:23:27.691 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2022-03-07T23:23:27.692 DEBUG:teuthology.orchestra.run.smithi078:> cat /etc/os-release 2022-03-07T23:23:27.741 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="Ubuntu" 2022-03-07T23:23:27.741 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-07T23:23:27.742 INFO:teuthology.orchestra.run.smithi078.stdout:ID=ubuntu 2022-03-07T23:23:27.742 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE=debian 2022-03-07T23:23:27.742 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-07T23:23:27.742 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="20.04" 2022-03-07T23:23:27.743 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-07T23:23:27.743 INFO:teuthology.orchestra.run.smithi078.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-07T23:23:27.743 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-07T23:23:27.743 INFO:teuthology.orchestra.run.smithi078.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-07T23:23:27.743 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_CODENAME=focal 2022-03-07T23:23:27.744 INFO:teuthology.orchestra.run.smithi078.stdout:UBUNTU_CODENAME=focal 2022-03-07T23:23:27.745 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2022-03-07T23:23:27.767 DEBUG:teuthology.orchestra.run.smithi166:> uname -m 2022-03-07T23:23:27.774 INFO:teuthology.orchestra.run.smithi166.stdout:x86_64 2022-03-07T23:23:27.774 DEBUG:teuthology.orchestra.run.smithi166:> cat /etc/os-release 2022-03-07T23:23:27.823 INFO:teuthology.orchestra.run.smithi166.stdout:NAME="Ubuntu" 2022-03-07T23:23:27.823 INFO:teuthology.orchestra.run.smithi166.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-07T23:23:27.823 INFO:teuthology.orchestra.run.smithi166.stdout:ID=ubuntu 2022-03-07T23:23:27.824 INFO:teuthology.orchestra.run.smithi166.stdout:ID_LIKE=debian 2022-03-07T23:23:27.824 INFO:teuthology.orchestra.run.smithi166.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-07T23:23:27.824 INFO:teuthology.orchestra.run.smithi166.stdout:VERSION_ID="20.04" 2022-03-07T23:23:27.825 INFO:teuthology.orchestra.run.smithi166.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-07T23:23:27.825 INFO:teuthology.orchestra.run.smithi166.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-07T23:23:27.825 INFO:teuthology.orchestra.run.smithi166.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-07T23:23:27.826 INFO:teuthology.orchestra.run.smithi166.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-07T23:23:27.826 INFO:teuthology.orchestra.run.smithi166.stdout:VERSION_CODENAME=focal 2022-03-07T23:23:27.826 INFO:teuthology.orchestra.run.smithi166.stdout:UBUNTU_CODENAME=focal 2022-03-07T23:23:27.827 INFO:teuthology.lock.ops:Updating smithi166.front.sepia.ceph.com on lock server 2022-03-07T23:23:27.845 DEBUG:teuthology.orchestra.run.smithi170:> uname -m 2022-03-07T23:23:27.853 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64 2022-03-07T23:23:27.853 DEBUG:teuthology.orchestra.run.smithi170:> cat /etc/os-release 2022-03-07T23:23:27.900 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="Ubuntu" 2022-03-07T23:23:27.901 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-03-07T23:23:27.901 INFO:teuthology.orchestra.run.smithi170.stdout:ID=ubuntu 2022-03-07T23:23:27.901 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE=debian 2022-03-07T23:23:27.901 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-03-07T23:23:27.901 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="20.04" 2022-03-07T23:23:27.902 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-07T23:23:27.902 INFO:teuthology.orchestra.run.smithi170.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-07T23:23:27.902 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-07T23:23:27.903 INFO:teuthology.orchestra.run.smithi170.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-07T23:23:27.903 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_CODENAME=focal 2022-03-07T23:23:27.903 INFO:teuthology.orchestra.run.smithi170.stdout:UBUNTU_CODENAME=focal 2022-03-07T23:23:27.904 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2022-03-07T23:23:27.922 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-03-07T23:23:27.936 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-03-07T23:23:27.981 INFO:teuthology.task.internal:Checking for old test directory... 2022-03-07T23:23:27.982 DEBUG:teuthology.orchestra.run.smithi078:> test '!' -e /home/ubuntu/cephtest 2022-03-07T23:23:27.985 DEBUG:teuthology.orchestra.run.smithi166:> test '!' -e /home/ubuntu/cephtest 2022-03-07T23:23:27.987 DEBUG:teuthology.orchestra.run.smithi170:> test '!' -e /home/ubuntu/cephtest 2022-03-07T23:23:27.993 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-03-07T23:23:28.015 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-03-07T23:23:28.016 DEBUG:teuthology.orchestra.run.smithi078:> test -z $(ls -A /var/lib/ceph) 2022-03-07T23:23:28.019 DEBUG:teuthology.orchestra.run.smithi166:> test -z $(ls -A /var/lib/ceph) 2022-03-07T23:23:28.021 DEBUG:teuthology.orchestra.run.smithi170:> test -z $(ls -A /var/lib/ceph) 2022-03-07T23:23:28.040 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-03-07T23:23:28.122 INFO:teuthology.run_tasks:Running task kernel... 2022-03-07T23:23:28.142 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-03-07T23:23:28.142 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-03-07T23:23:28.143 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}, 'host.b': {'hwe': True}, 'client.b': {'hwe': True}, 'host.c': {'hwe': True}, 'client.c': {'hwe': True}} 2022-03-07T23:23:28.143 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.b': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.c': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-03-07T23:23:28.144 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2022-03-07T23:23:28.145 DEBUG:teuthology.orchestra.run.smithi166:> uname -r 2022-03-07T23:23:28.145 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2022-03-07T23:23:28.152 INFO:teuthology.orchestra.run.smithi166.stdout:5.4.0-97-generic 2022-03-07T23:23:28.152 INFO:teuthology.task.kernel:Running kernel on smithi166: 5.4.0-97-generic 2022-03-07T23:23:28.153 DEBUG:teuthology.orchestra.run.smithi166:> sudo apt-get clean 2022-03-07T23:23:28.155 INFO:teuthology.orchestra.run.smithi170.stdout:5.4.0-97-generic 2022-03-07T23:23:28.155 INFO:teuthology.orchestra.run.smithi078.stdout:5.4.0-97-generic 2022-03-07T23:23:28.155 INFO:teuthology.task.kernel:Running kernel on smithi170: 5.4.0-97-generic 2022-03-07T23:23:28.156 DEBUG:teuthology.orchestra.run.smithi170:> sudo apt-get clean 2022-03-07T23:23:28.156 INFO:teuthology.task.kernel:Running kernel on smithi078: 5.4.0-97-generic 2022-03-07T23:23:28.157 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get clean 2022-03-07T23:23:28.241 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get update 2022-03-07T23:23:28.243 DEBUG:teuthology.orchestra.run.smithi170:> sudo apt-get update 2022-03-07T23:23:28.257 DEBUG:teuthology.orchestra.run.smithi166:> sudo apt-get update 2022-03-07T23:23:28.387 INFO:teuthology.orchestra.run.smithi078.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-07T23:23:28.388 INFO:teuthology.orchestra.run.smithi170.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-07T23:23:28.497 INFO:teuthology.orchestra.run.smithi170.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-07T23:23:28.503 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-07T23:23:28.564 INFO:teuthology.orchestra.run.smithi166.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-07T23:23:28.564 INFO:teuthology.orchestra.run.smithi166.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-03-07T23:23:28.579 INFO:teuthology.orchestra.run.smithi170.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-07T23:23:28.586 INFO:teuthology.orchestra.run.smithi078.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-07T23:23:28.622 INFO:teuthology.orchestra.run.smithi166.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-03-07T23:23:28.799 INFO:teuthology.orchestra.run.smithi170.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,275 kB] 2022-03-07T23:23:28.799 INFO:teuthology.orchestra.run.smithi078.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [387 kB] 2022-03-07T23:23:28.799 INFO:teuthology.orchestra.run.smithi078.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,275 kB] 2022-03-07T23:23:28.832 INFO:teuthology.orchestra.run.smithi170.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [387 kB] 2022-03-07T23:23:28.842 INFO:teuthology.orchestra.run.smithi170.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [224 kB] 2022-03-07T23:23:28.847 INFO:teuthology.orchestra.run.smithi170.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-07T23:23:28.847 INFO:teuthology.orchestra.run.smithi170.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [773 kB] 2022-03-07T23:23:28.856 INFO:teuthology.orchestra.run.smithi078.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [224 kB] 2022-03-07T23:23:28.863 INFO:teuthology.orchestra.run.smithi078.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-07T23:23:28.863 INFO:teuthology.orchestra.run.smithi078.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-07T23:23:28.865 INFO:teuthology.orchestra.run.smithi078.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [773 kB] 2022-03-07T23:23:28.884 INFO:teuthology.orchestra.run.smithi078.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [110 kB] 2022-03-07T23:23:28.885 INFO:teuthology.orchestra.run.smithi170.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-07T23:23:28.886 INFO:teuthology.orchestra.run.smithi170.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [110 kB] 2022-03-07T23:23:28.889 INFO:teuthology.orchestra.run.smithi170.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-07T23:23:28.889 INFO:teuthology.orchestra.run.smithi170.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-07T23:23:28.889 INFO:teuthology.orchestra.run.smithi078.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-07T23:23:28.890 INFO:teuthology.orchestra.run.smithi078.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-07T23:23:28.903 INFO:teuthology.orchestra.run.smithi170.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [680 kB] 2022-03-07T23:23:28.923 INFO:teuthology.orchestra.run.smithi170.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-07T23:23:28.928 INFO:teuthology.orchestra.run.smithi078.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [680 kB] 2022-03-07T23:23:28.939 INFO:teuthology.orchestra.run.smithi078.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-07T23:23:28.942 INFO:teuthology.orchestra.run.smithi078.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-07T23:23:28.944 INFO:teuthology.orchestra.run.smithi170.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-07T23:23:28.953 INFO:teuthology.orchestra.run.smithi170.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-07T23:23:28.954 INFO:teuthology.orchestra.run.smithi170.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-07T23:23:28.954 INFO:teuthology.orchestra.run.smithi170.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-07T23:23:28.954 INFO:teuthology.orchestra.run.smithi170.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-07T23:23:28.960 INFO:teuthology.orchestra.run.smithi078.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-07T23:23:28.961 INFO:teuthology.orchestra.run.smithi078.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-07T23:23:28.962 INFO:teuthology.orchestra.run.smithi078.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-07T23:23:28.962 INFO:teuthology.orchestra.run.smithi078.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-07T23:23:28.965 INFO:teuthology.orchestra.run.smithi170.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-07T23:23:28.974 INFO:teuthology.orchestra.run.smithi078.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-07T23:23:29.025 INFO:teuthology.orchestra.run.smithi166.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-03-07T23:23:29.076 INFO:teuthology.orchestra.run.smithi166.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,275 kB] 2022-03-07T23:23:29.103 INFO:teuthology.orchestra.run.smithi078.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,608 kB] 2022-03-07T23:23:29.115 INFO:teuthology.orchestra.run.smithi170.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,608 kB] 2022-03-07T23:23:29.215 INFO:teuthology.orchestra.run.smithi166.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [610 kB] 2022-03-07T23:23:29.428 INFO:teuthology.orchestra.run.smithi170.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [610 kB] 2022-03-07T23:23:29.467 INFO:teuthology.orchestra.run.smithi170.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [308 kB] 2022-03-07T23:23:29.482 INFO:teuthology.orchestra.run.smithi170.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-07T23:23:29.483 INFO:teuthology.orchestra.run.smithi170.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-07T23:23:29.483 INFO:teuthology.orchestra.run.smithi170.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [827 kB] 2022-03-07T23:23:29.492 INFO:teuthology.orchestra.run.smithi078.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [610 kB] 2022-03-07T23:23:29.516 INFO:teuthology.orchestra.run.smithi078.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [308 kB] 2022-03-07T23:23:29.520 INFO:teuthology.orchestra.run.smithi170.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [118 kB] 2022-03-07T23:23:29.524 INFO:teuthology.orchestra.run.smithi170.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-07T23:23:29.525 INFO:teuthology.orchestra.run.smithi170.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [671 kB] 2022-03-07T23:23:29.528 INFO:teuthology.orchestra.run.smithi166.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [387 kB] 2022-03-07T23:23:29.533 INFO:teuthology.orchestra.run.smithi078.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-07T23:23:29.534 INFO:teuthology.orchestra.run.smithi078.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-07T23:23:29.535 INFO:teuthology.orchestra.run.smithi078.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [827 kB] 2022-03-07T23:23:29.546 INFO:teuthology.orchestra.run.smithi166.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,608 kB] 2022-03-07T23:23:29.546 INFO:teuthology.orchestra.run.smithi170.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [908 kB] 2022-03-07T23:23:29.552 INFO:teuthology.orchestra.run.smithi166.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [224 kB] 2022-03-07T23:23:29.565 INFO:teuthology.orchestra.run.smithi166.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [9,732 B] 2022-03-07T23:23:29.566 INFO:teuthology.orchestra.run.smithi166.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [21.7 kB] 2022-03-07T23:23:29.567 INFO:teuthology.orchestra.run.smithi166.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [773 kB] 2022-03-07T23:23:29.579 INFO:teuthology.orchestra.run.smithi170.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [201 kB] 2022-03-07T23:23:29.596 INFO:teuthology.orchestra.run.smithi078.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [118 kB] 2022-03-07T23:23:29.602 INFO:teuthology.orchestra.run.smithi078.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-07T23:23:29.603 INFO:teuthology.orchestra.run.smithi078.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [908 kB] 2022-03-07T23:23:29.615 INFO:teuthology.orchestra.run.smithi170.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-07T23:23:29.616 INFO:teuthology.orchestra.run.smithi170.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,464 B] 2022-03-07T23:23:29.617 INFO:teuthology.orchestra.run.smithi170.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-07T23:23:29.617 INFO:teuthology.orchestra.run.smithi170.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-07T23:23:29.629 INFO:teuthology.orchestra.run.smithi166.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [110 kB] 2022-03-07T23:23:29.631 INFO:teuthology.orchestra.run.smithi078.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [671 kB] 2022-03-07T23:23:29.633 INFO:teuthology.orchestra.run.smithi166.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [504 B] 2022-03-07T23:23:29.633 INFO:teuthology.orchestra.run.smithi166.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [680 kB] 2022-03-07T23:23:29.654 INFO:teuthology.orchestra.run.smithi078.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [201 kB] 2022-03-07T23:23:29.659 INFO:teuthology.orchestra.run.smithi078.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-07T23:23:29.665 INFO:teuthology.orchestra.run.smithi166.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [536 kB] 2022-03-07T23:23:29.677 INFO:teuthology.orchestra.run.smithi170.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-07T23:23:29.678 INFO:teuthology.orchestra.run.smithi170.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-07T23:23:29.679 INFO:teuthology.orchestra.run.smithi170.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-07T23:23:29.680 INFO:teuthology.orchestra.run.smithi170.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-07T23:23:29.680 INFO:teuthology.orchestra.run.smithi170.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-07T23:23:29.687 INFO:teuthology.orchestra.run.smithi166.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [117 kB] 2022-03-07T23:23:29.692 INFO:teuthology.orchestra.run.smithi166.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [13.2 kB] 2022-03-07T23:23:29.692 INFO:teuthology.orchestra.run.smithi078.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-07T23:23:29.693 INFO:teuthology.orchestra.run.smithi078.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,464 B] 2022-03-07T23:23:29.694 INFO:teuthology.orchestra.run.smithi078.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-07T23:23:29.694 INFO:teuthology.orchestra.run.smithi078.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-07T23:23:29.695 INFO:teuthology.orchestra.run.smithi078.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-07T23:23:29.700 INFO:teuthology.orchestra.run.smithi170.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-07T23:23:29.702 INFO:teuthology.orchestra.run.smithi078.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-07T23:23:29.712 INFO:teuthology.orchestra.run.smithi166.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [308 kB] 2022-03-07T23:23:29.733 INFO:teuthology.orchestra.run.smithi166.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,180 B] 2022-03-07T23:23:29.733 INFO:teuthology.orchestra.run.smithi166.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [20.7 kB] 2022-03-07T23:23:29.734 INFO:teuthology.orchestra.run.smithi166.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,196 B] 2022-03-07T23:23:29.734 INFO:teuthology.orchestra.run.smithi166.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [500 B] 2022-03-07T23:23:29.755 INFO:teuthology.orchestra.run.smithi166.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [14.8 kB] 2022-03-07T23:23:29.756 INFO:teuthology.orchestra.run.smithi166.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [827 kB] 2022-03-07T23:23:29.782 INFO:teuthology.orchestra.run.smithi170.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-07T23:23:29.782 INFO:teuthology.orchestra.run.smithi170.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-07T23:23:29.785 INFO:teuthology.orchestra.run.smithi078.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-07T23:23:29.786 INFO:teuthology.orchestra.run.smithi078.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-07T23:23:29.787 INFO:teuthology.orchestra.run.smithi078.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-07T23:23:29.787 INFO:teuthology.orchestra.run.smithi078.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-07T23:23:29.788 INFO:teuthology.orchestra.run.smithi078.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-07T23:23:29.791 INFO:teuthology.orchestra.run.smithi166.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [23.1 kB] 2022-03-07T23:23:29.793 INFO:teuthology.orchestra.run.smithi166.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [118 kB] 2022-03-07T23:23:29.798 INFO:teuthology.orchestra.run.smithi166.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [500 B] 2022-03-07T23:23:29.798 INFO:teuthology.orchestra.run.smithi166.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [671 kB] 2022-03-07T23:23:29.817 INFO:teuthology.orchestra.run.smithi166.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [908 kB] 2022-03-07T23:23:29.851 INFO:teuthology.orchestra.run.smithi166.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [201 kB] 2022-03-07T23:23:29.856 INFO:teuthology.orchestra.run.smithi166.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.2 kB] 2022-03-07T23:23:29.880 INFO:teuthology.orchestra.run.smithi166.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,464 B] 2022-03-07T23:23:29.881 INFO:teuthology.orchestra.run.smithi166.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [23.8 kB] 2022-03-07T23:23:29.881 INFO:teuthology.orchestra.run.smithi166.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,312 B] 2022-03-07T23:23:29.882 INFO:teuthology.orchestra.run.smithi166.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [580 B] 2022-03-07T23:23:29.939 INFO:teuthology.orchestra.run.smithi166.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [42.2 kB] 2022-03-07T23:23:29.940 INFO:teuthology.orchestra.run.smithi166.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [34.8 kB] 2022-03-07T23:23:29.941 INFO:teuthology.orchestra.run.smithi166.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [10.1 kB] 2022-03-07T23:23:29.942 INFO:teuthology.orchestra.run.smithi166.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [22.7 kB] 2022-03-07T23:23:29.942 INFO:teuthology.orchestra.run.smithi166.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [12.9 kB] 2022-03-07T23:23:29.962 INFO:teuthology.orchestra.run.smithi166.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [15.4 kB] 2022-03-07T23:23:30.044 INFO:teuthology.orchestra.run.smithi166.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [804 B] 2022-03-07T23:23:31.963 INFO:teuthology.orchestra.run.smithi170.stdout:Fetched 10.0 MB in 2s (5,833 kB/s) 2022-03-07T23:23:31.987 INFO:teuthology.orchestra.run.smithi078.stdout:Fetched 10.0 MB in 2s (5,906 kB/s) 2022-03-07T23:23:32.733 INFO:teuthology.orchestra.run.smithi166.stdout:Fetched 10.0 MB in 2s (4,494 kB/s) 2022-03-07T23:23:33.146 INFO:teuthology.orchestra.run.smithi170.stdout:Reading package lists... 2022-03-07T23:23:33.170 DEBUG:teuthology.orchestra.run.smithi170:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-07T23:23:33.195 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2022-03-07T23:23:33.220 DEBUG:teuthology.orchestra.run.smithi078:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-07T23:23:33.243 INFO:teuthology.orchestra.run.smithi170.stdout:Reading package lists... 2022-03-07T23:23:33.291 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2022-03-07T23:23:33.417 INFO:teuthology.orchestra.run.smithi170.stdout:Building dependency tree... 2022-03-07T23:23:33.417 INFO:teuthology.orchestra.run.smithi170.stdout:Reading state information... 2022-03-07T23:23:33.464 INFO:teuthology.orchestra.run.smithi078.stdout:Building dependency tree... 2022-03-07T23:23:33.464 INFO:teuthology.orchestra.run.smithi078.stdout:Reading state information... 2022-03-07T23:23:33.558 INFO:teuthology.orchestra.run.smithi170.stdout:The following additional packages will be installed: 2022-03-07T23:23:33.559 INFO:teuthology.orchestra.run.smithi170.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-07T23:23:33.559 INFO:teuthology.orchestra.run.smithi170.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-modules-5.13.0-30-generic 2022-03-07T23:23:33.559 INFO:teuthology.orchestra.run.smithi170.stdout: linux-modules-extra-5.13.0-30-generic thermald upower usbmuxd 2022-03-07T23:23:33.560 INFO:teuthology.orchestra.run.smithi170.stdout:Suggested packages: 2022-03-07T23:23:33.560 INFO:teuthology.orchestra.run.smithi170.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-07T23:23:33.560 INFO:teuthology.orchestra.run.smithi170.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-30-generic 2022-03-07T23:23:33.603 INFO:teuthology.orchestra.run.smithi170.stdout:The following NEW packages will be installed: 2022-03-07T23:23:33.603 INFO:teuthology.orchestra.run.smithi170.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-07T23:23:33.603 INFO:teuthology.orchestra.run.smithi170.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-image-generic-hwe-20.04 2022-03-07T23:23:33.604 INFO:teuthology.orchestra.run.smithi170.stdout: linux-modules-5.13.0-30-generic linux-modules-extra-5.13.0-30-generic 2022-03-07T23:23:33.604 INFO:teuthology.orchestra.run.smithi170.stdout: thermald upower usbmuxd 2022-03-07T23:23:33.612 INFO:teuthology.orchestra.run.smithi078.stdout:The following additional packages will be installed: 2022-03-07T23:23:33.612 INFO:teuthology.orchestra.run.smithi078.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-07T23:23:33.613 INFO:teuthology.orchestra.run.smithi078.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-modules-5.13.0-30-generic 2022-03-07T23:23:33.613 INFO:teuthology.orchestra.run.smithi078.stdout: linux-modules-extra-5.13.0-30-generic thermald upower usbmuxd 2022-03-07T23:23:33.614 INFO:teuthology.orchestra.run.smithi078.stdout:Suggested packages: 2022-03-07T23:23:33.614 INFO:teuthology.orchestra.run.smithi078.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-07T23:23:33.614 INFO:teuthology.orchestra.run.smithi078.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-30-generic 2022-03-07T23:23:33.662 INFO:teuthology.orchestra.run.smithi078.stdout:The following NEW packages will be installed: 2022-03-07T23:23:33.663 INFO:teuthology.orchestra.run.smithi078.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-07T23:23:33.663 INFO:teuthology.orchestra.run.smithi078.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-image-generic-hwe-20.04 2022-03-07T23:23:33.664 INFO:teuthology.orchestra.run.smithi078.stdout: linux-modules-5.13.0-30-generic linux-modules-extra-5.13.0-30-generic 2022-03-07T23:23:33.664 INFO:teuthology.orchestra.run.smithi078.stdout: thermald upower usbmuxd 2022-03-07T23:23:33.847 INFO:teuthology.orchestra.run.smithi078.stdout:0 upgraded, 13 newly installed, 0 to remove and 88 not upgraded. 2022-03-07T23:23:33.847 INFO:teuthology.orchestra.run.smithi078.stdout:Need to get 80.1 MB of archives. 2022-03-07T23:23:33.847 INFO:teuthology.orchestra.run.smithi078.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-07T23:23:33.848 INFO:teuthology.orchestra.run.smithi078.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-07T23:23:33.937 INFO:teuthology.orchestra.run.smithi166.stdout:Reading package lists... 2022-03-07T23:23:33.960 DEBUG:teuthology.orchestra.run.smithi166:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-07T23:23:34.030 INFO:teuthology.orchestra.run.smithi166.stdout:Reading package lists... 2022-03-07T23:23:34.129 INFO:teuthology.orchestra.run.smithi170.stdout:0 upgraded, 13 newly installed, 0 to remove and 88 not upgraded. 2022-03-07T23:23:34.129 INFO:teuthology.orchestra.run.smithi170.stdout:Need to get 80.1 MB of archives. 2022-03-07T23:23:34.129 INFO:teuthology.orchestra.run.smithi170.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-07T23:23:34.130 INFO:teuthology.orchestra.run.smithi170.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-07T23:23:34.132 INFO:teuthology.orchestra.run.smithi078.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-07T23:23:34.156 INFO:teuthology.orchestra.run.smithi078.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-07T23:23:34.157 INFO:teuthology.orchestra.run.smithi078.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-07T23:23:34.204 INFO:teuthology.orchestra.run.smithi166.stdout:Building dependency tree... 2022-03-07T23:23:34.205 INFO:teuthology.orchestra.run.smithi166.stdout:Reading state information... 2022-03-07T23:23:34.242 INFO:teuthology.orchestra.run.smithi078.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-07T23:23:34.261 INFO:teuthology.orchestra.run.smithi078.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [18.3 MB] 2022-03-07T23:23:34.346 INFO:teuthology.orchestra.run.smithi166.stdout:The following additional packages will be installed: 2022-03-07T23:23:34.346 INFO:teuthology.orchestra.run.smithi166.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-07T23:23:34.346 INFO:teuthology.orchestra.run.smithi166.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-modules-5.13.0-30-generic 2022-03-07T23:23:34.347 INFO:teuthology.orchestra.run.smithi166.stdout: linux-modules-extra-5.13.0-30-generic thermald upower usbmuxd 2022-03-07T23:23:34.347 INFO:teuthology.orchestra.run.smithi166.stdout:Suggested packages: 2022-03-07T23:23:34.347 INFO:teuthology.orchestra.run.smithi166.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.13-source-5.13.0 2022-03-07T23:23:34.348 INFO:teuthology.orchestra.run.smithi166.stdout: linux-hwe-5.13-tools linux-headers-5.13.0-30-generic 2022-03-07T23:23:34.398 INFO:teuthology.orchestra.run.smithi166.stdout:The following NEW packages will be installed: 2022-03-07T23:23:34.399 INFO:teuthology.orchestra.run.smithi166.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-03-07T23:23:34.399 INFO:teuthology.orchestra.run.smithi166.stdout: libusbmuxd6 linux-image-5.13.0-30-generic linux-image-generic-hwe-20.04 2022-03-07T23:23:34.400 INFO:teuthology.orchestra.run.smithi166.stdout: linux-modules-5.13.0-30-generic linux-modules-extra-5.13.0-30-generic 2022-03-07T23:23:34.400 INFO:teuthology.orchestra.run.smithi166.stdout: thermald upower usbmuxd 2022-03-07T23:23:34.580 INFO:teuthology.orchestra.run.smithi166.stdout:0 upgraded, 13 newly installed, 0 to remove and 88 not upgraded. 2022-03-07T23:23:34.581 INFO:teuthology.orchestra.run.smithi166.stdout:Need to get 80.1 MB of archives. 2022-03-07T23:23:34.581 INFO:teuthology.orchestra.run.smithi166.stdout:After this operation, 409 MB of additional disk space will be used. 2022-03-07T23:23:34.581 INFO:teuthology.orchestra.run.smithi166.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-03-07T23:23:34.853 INFO:teuthology.orchestra.run.smithi166.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-07T23:23:34.877 INFO:teuthology.orchestra.run.smithi166.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-07T23:23:34.878 INFO:teuthology.orchestra.run.smithi166.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-07T23:23:34.960 INFO:teuthology.orchestra.run.smithi166.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-07T23:23:34.972 INFO:teuthology.orchestra.run.smithi166.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [18.3 MB] 2022-03-07T23:23:35.028 INFO:teuthology.orchestra.run.smithi170.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-03-07T23:23:35.103 INFO:teuthology.orchestra.run.smithi078.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [10.0 MB] 2022-03-07T23:23:35.140 INFO:teuthology.orchestra.run.smithi170.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-03-07T23:23:35.216 INFO:teuthology.orchestra.run.smithi170.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-03-07T23:23:35.363 INFO:teuthology.orchestra.run.smithi078.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [51.1 MB] 2022-03-07T23:23:35.570 INFO:teuthology.orchestra.run.smithi170.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-03-07T23:23:35.575 INFO:teuthology.orchestra.run.smithi170.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [18.3 MB] 2022-03-07T23:23:35.832 INFO:teuthology.orchestra.run.smithi166.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [10.0 MB] 2022-03-07T23:23:36.093 INFO:teuthology.orchestra.run.smithi166.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [51.1 MB] 2022-03-07T23:23:37.023 INFO:teuthology.orchestra.run.smithi170.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [10.0 MB] 2022-03-07T23:23:37.120 INFO:teuthology.orchestra.run.smithi078.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.30.33~20.04.17 [2,616 B] 2022-03-07T23:23:37.121 INFO:teuthology.orchestra.run.smithi078.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-07T23:23:37.121 INFO:teuthology.orchestra.run.smithi078.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-07T23:23:37.126 INFO:teuthology.orchestra.run.smithi078.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-07T23:23:37.128 INFO:teuthology.orchestra.run.smithi078.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-07T23:23:37.529 INFO:teuthology.orchestra.run.smithi170.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.13.0-30-generic amd64 5.13.0-30.33~20.04.1 [51.1 MB] 2022-03-07T23:23:37.534 INFO:teuthology.orchestra.run.smithi078.stdout:Fetched 80.1 MB in 3s (23.2 MB/s) 2022-03-07T23:23:37.760 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-07T23:23:37.803 INFO:teuthology.orchestra.run.smithi166.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.30.33~20.04.17 [2,616 B] 2022-03-07T23:23:37.803 INFO:teuthology.orchestra.run.smithi166.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-07T23:23:37.804 INFO:teuthology.orchestra.run.smithi166.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-07T23:23:37.809 INFO:teuthology.orchestra.run.smithi166.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-07T23:23:37.811 INFO:teuthology.orchestra.run.smithi166.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-07T23:23:38.243 INFO:teuthology.orchestra.run.smithi166.stdout:Fetched 80.1 MB in 3s (23.5 MB/s) 2022-03-07T23:23:38.454 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-07T23:23:39.245 INFO:teuthology.orchestra.run.smithi078.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-07T23:23:39.250 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-07T23:23:39.377 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-07T23:23:39.771 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-07T23:23:39.784 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-07T23:23:39.821 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-07T23:23:39.941 INFO:teuthology.orchestra.run.smithi166.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-07T23:23:39.945 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-07T23:23:40.101 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-07T23:23:40.141 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-07T23:23:40.153 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-07T23:23:40.183 INFO:teuthology.orchestra.run.smithi170.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.13.0.30.33~20.04.17 [2,616 B] 2022-03-07T23:23:40.190 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-07T23:23:40.265 INFO:teuthology.orchestra.run.smithi170.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.1 [31.6 kB] 2022-03-07T23:23:40.429 INFO:teuthology.orchestra.run.smithi170.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-03-07T23:23:40.479 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-07T23:23:40.492 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-07T23:23:40.502 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-07T23:23:40.515 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-07T23:23:40.537 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-07T23:23:40.552 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-07T23:23:40.756 INFO:teuthology.orchestra.run.smithi170.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-03-07T23:23:40.890 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-07T23:23:40.901 INFO:teuthology.orchestra.run.smithi170.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-03-07T23:23:40.902 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-07T23:23:40.921 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-07T23:23:40.935 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-07T23:23:40.939 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-07T23:23:40.979 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-07T23:23:41.242 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-07T23:23:41.258 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-07T23:23:41.292 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-07T23:23:41.299 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-modules-5.13.0-30-generic. 2022-03-07T23:23:41.312 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../05-linux-modules-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-07T23:23:41.315 INFO:teuthology.orchestra.run.smithi170.stdout:Fetched 80.1 MB in 7s (11.0 MB/s) 2022-03-07T23:23:41.349 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T23:23:41.551 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-03-07T23:23:41.654 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-07T23:23:41.667 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-07T23:23:41.703 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-07T23:23:42.029 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package linux-modules-5.13.0-30-generic. 2022-03-07T23:23:42.045 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../05-linux-modules-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-07T23:23:42.089 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T23:23:42.992 INFO:teuthology.orchestra.run.smithi170.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102042 files and directories currently installed.) 2022-03-07T23:23:42.997 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-03-07T23:23:43.106 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-07T23:23:43.467 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package libplist3:amd64. 2022-03-07T23:23:43.482 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-03-07T23:23:43.499 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-image-5.13.0-30-generic. 2022-03-07T23:23:43.512 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../06-linux-image-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-07T23:23:43.525 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-03-07T23:23:43.598 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T23:23:43.861 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-03-07T23:23:43.876 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-03-07T23:23:43.921 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-07T23:23:44.077 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-30-generic. 2022-03-07T23:23:44.093 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-07T23:23:44.135 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T23:23:44.215 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-03-07T23:23:44.228 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-03-07T23:23:44.272 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-07T23:23:44.389 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package linux-image-5.13.0-30-generic. 2022-03-07T23:23:44.406 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../06-linux-image-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-07T23:23:44.495 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T23:23:44.609 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-03-07T23:23:44.622 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-03-07T23:23:44.667 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-07T23:23:44.978 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-30-generic. 2022-03-07T23:23:44.979 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package linux-modules-5.13.0-30-generic. 2022-03-07T23:23:44.989 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-07T23:23:44.994 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../05-linux-modules-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-07T23:23:45.034 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T23:23:45.036 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T23:23:47.437 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package linux-image-5.13.0-30-generic. 2022-03-07T23:23:47.450 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../06-linux-image-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-07T23:23:47.554 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T23:23:48.065 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package linux-modules-extra-5.13.0-30-generic. 2022-03-07T23:23:48.078 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../07-linux-modules-extra-5.13.0-30-generic_5.13.0-30.33~20.04.1_amd64.deb ... 2022-03-07T23:23:48.123 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T23:23:50.302 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-07T23:23:50.555 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.30.33~20.04.17_amd64.deb ... 2022-03-07T23:23:50.556 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-07T23:23:50.711 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-07T23:23:50.731 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-07T23:23:50.770 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-07T23:23:50.922 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-07T23:23:50.937 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.30.33~20.04.17_amd64.deb ... 2022-03-07T23:23:50.980 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-07T23:23:51.156 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package thermald. 2022-03-07T23:23:51.171 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-07T23:23:51.215 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-07T23:23:51.317 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-07T23:23:51.330 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-07T23:23:51.367 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-07T23:23:51.745 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package thermald. 2022-03-07T23:23:51.760 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-07T23:23:51.773 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package upower. 2022-03-07T23:23:51.787 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-07T23:23:51.795 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-07T23:23:51.841 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-07T23:23:52.250 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package usbmuxd. 2022-03-07T23:23:52.266 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-07T23:23:52.306 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-07T23:23:52.336 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package upower. 2022-03-07T23:23:52.350 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-07T23:23:52.405 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-07T23:23:52.589 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-07T23:23:52.711 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-07T23:23:52.812 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package usbmuxd. 2022-03-07T23:23:52.826 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-07T23:23:52.870 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-07T23:23:52.888 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-07T23:23:53.014 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-07T23:23:53.140 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-07T23:23:53.167 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-07T23:23:53.266 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-07T23:23:53.291 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-07T23:23:53.409 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-07T23:23:53.417 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-07T23:23:53.544 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-07T23:23:53.663 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-07T23:23:53.787 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-07T23:23:53.858 INFO:teuthology.orchestra.run.smithi078.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-07T23:23:53.910 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-07T23:23:53.930 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-07T23:23:54.023 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-03-07T23:23:54.039 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.13.0.30.33~20.04.17_amd64.deb ... 2022-03-07T23:23:54.081 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-07T23:23:54.121 INFO:teuthology.orchestra.run.smithi078.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-07T23:23:54.133 INFO:teuthology.orchestra.run.smithi078.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-07T23:23:54.134 INFO:teuthology.orchestra.run.smithi078.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-07T23:23:54.393 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package libevdev2:amd64. 2022-03-07T23:23:54.409 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.1_amd64.deb ... 2022-03-07T23:23:54.419 INFO:teuthology.orchestra.run.smithi166.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-07T23:23:54.451 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-07T23:23:54.481 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-07T23:23:54.611 INFO:teuthology.orchestra.run.smithi078.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-07T23:23:54.676 INFO:teuthology.orchestra.run.smithi166.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-07T23:23:54.684 INFO:teuthology.orchestra.run.smithi166.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-07T23:23:54.685 INFO:teuthology.orchestra.run.smithi166.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-07T23:23:54.763 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package thermald. 2022-03-07T23:23:54.778 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-03-07T23:23:54.821 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-03-07T23:23:54.834 INFO:teuthology.orchestra.run.smithi078.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-07T23:23:54.882 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-07T23:23:55.182 INFO:teuthology.orchestra.run.smithi166.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-07T23:23:55.183 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-07T23:23:55.184 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-07T23:23:55.296 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package upower. 2022-03-07T23:23:55.309 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-03-07T23:23:55.373 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking upower (0.99.11-1build2) ... 2022-03-07T23:23:55.415 INFO:teuthology.orchestra.run.smithi166.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-07T23:23:55.461 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-07T23:23:55.678 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T23:23:55.738 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package usbmuxd. 2022-03-07T23:23:55.753 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-03-07T23:23:55.781 INFO:teuthology.orchestra.run.smithi166.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-07T23:23:55.782 INFO:teuthology.orchestra.run.smithi166.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-07T23:23:55.788 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-07T23:23:56.071 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-03-07T23:23:56.201 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-03-07T23:23:56.291 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T23:23:56.327 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-03-07T23:23:56.453 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-03-07T23:23:56.579 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-03-07T23:23:56.713 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.1) ... 2022-03-07T23:23:56.841 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up upower (0.99.11-1build2) ... 2022-03-07T23:23:56.972 INFO:teuthology.orchestra.run.smithi078.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-30-generic 2022-03-07T23:23:56.973 INFO:teuthology.orchestra.run.smithi078.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-30-generic 2022-03-07T23:23:57.139 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T23:23:57.278 INFO:teuthology.orchestra.run.smithi170.stdout:upower.service is a disabled or a static unit, not starting it. 2022-03-07T23:23:57.349 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-03-07T23:23:57.541 INFO:teuthology.orchestra.run.smithi166.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-30-generic 2022-03-07T23:23:57.542 INFO:teuthology.orchestra.run.smithi166.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-30-generic 2022-03-07T23:23:57.587 INFO:teuthology.orchestra.run.smithi170.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-03-07T23:23:57.597 INFO:teuthology.orchestra.run.smithi170.stdout:Adding system user `usbmux' (UID 114) ... 2022-03-07T23:23:57.598 INFO:teuthology.orchestra.run.smithi170.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-03-07T23:23:57.727 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T23:23:58.159 INFO:teuthology.orchestra.run.smithi170.stdout:Not creating home directory `/var/lib/usbmux'. 2022-03-07T23:23:58.371 INFO:teuthology.orchestra.run.smithi170.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-03-07T23:23:58.531 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-03-07T23:23:58.599 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T23:23:58.843 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-03-07T23:23:58.843 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-03-07T23:23:59.037 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T23:23:59.302 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T23:23:59.909 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-07T23:24:00.035 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-07T23:24:00.561 INFO:teuthology.orchestra.run.smithi170.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.13.0-30-generic 2022-03-07T23:24:00.562 INFO:teuthology.orchestra.run.smithi170.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.13.0-30-generic 2022-03-07T23:24:00.597 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-07T23:24:00.721 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up linux-modules-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T23:24:00.723 INFO:teuthology.orchestra.run.smithi166.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-07T23:24:02.047 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up linux-modules-extra-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T23:24:03.374 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up linux-image-generic-hwe-20.04 (5.13.0.30.33~20.04.17) ... 2022-03-07T23:24:03.492 INFO:teuthology.orchestra.run.smithi170.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2022-03-07T23:24:04.662 INFO:teuthology.orchestra.run.smithi166.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-07T23:24:04.824 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-07T23:24:05.476 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-07T23:24:05.593 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T23:24:05.793 INFO:teuthology.orchestra.run.smithi078.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-07T23:24:05.794 INFO:teuthology.orchestra.run.smithi078.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-30-generic 2022-03-07T23:24:06.072 INFO:teuthology.orchestra.run.smithi166.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-07T23:24:06.222 INFO:teuthology.orchestra.run.smithi166.stdout:Processing triggers for linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T23:24:06.428 INFO:teuthology.orchestra.run.smithi166.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-07T23:24:06.428 INFO:teuthology.orchestra.run.smithi166.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-30-generic 2022-03-07T23:24:07.148 INFO:teuthology.orchestra.run.smithi170.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-07T23:24:07.841 INFO:teuthology.orchestra.run.smithi170.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.1) ... 2022-03-07T23:24:08.467 INFO:teuthology.orchestra.run.smithi170.stdout:Processing triggers for linux-image-5.13.0-30-generic (5.13.0-30.33~20.04.1) ... 2022-03-07T23:24:08.690 INFO:teuthology.orchestra.run.smithi170.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-03-07T23:24:08.691 INFO:teuthology.orchestra.run.smithi170.stdout:update-initramfs: Generating /boot/initrd.img-5.13.0-30-generic 2022-03-07T23:24:29.259 INFO:teuthology.orchestra.run.smithi078.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-07T23:24:29.260 INFO:teuthology.orchestra.run.smithi078.stdout:Sourcing file `/etc/default/grub' 2022-03-07T23:24:29.268 INFO:teuthology.orchestra.run.smithi078.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-07T23:24:29.296 INFO:teuthology.orchestra.run.smithi078.stdout:Generating grub configuration file ... 2022-03-07T23:24:29.634 INFO:teuthology.orchestra.run.smithi078.stdout:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-07T23:24:29.658 INFO:teuthology.orchestra.run.smithi078.stdout:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-07T23:24:29.972 INFO:teuthology.orchestra.run.smithi166.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-07T23:24:29.972 INFO:teuthology.orchestra.run.smithi166.stdout:Sourcing file `/etc/default/grub' 2022-03-07T23:24:29.983 INFO:teuthology.orchestra.run.smithi166.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-07T23:24:30.009 INFO:teuthology.orchestra.run.smithi166.stdout:Generating grub configuration file ... 2022-03-07T23:24:30.082 INFO:teuthology.orchestra.run.smithi078.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-07T23:24:30.093 INFO:teuthology.orchestra.run.smithi078.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-07T23:24:30.432 INFO:teuthology.orchestra.run.smithi166.stdout:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-07T23:24:30.460 INFO:teuthology.orchestra.run.smithi166.stdout:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-07T23:24:30.867 INFO:teuthology.orchestra.run.smithi166.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-07T23:24:30.878 INFO:teuthology.orchestra.run.smithi166.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-07T23:24:32.067 INFO:teuthology.orchestra.run.smithi170.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-03-07T23:24:32.067 INFO:teuthology.orchestra.run.smithi170.stdout:Sourcing file `/etc/default/grub' 2022-03-07T23:24:32.082 INFO:teuthology.orchestra.run.smithi170.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-07T23:24:32.103 INFO:teuthology.orchestra.run.smithi170.stdout:Generating grub configuration file ... 2022-03-07T23:24:32.485 INFO:teuthology.orchestra.run.smithi170.stdout:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-07T23:24:32.518 INFO:teuthology.orchestra.run.smithi170.stdout:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-07T23:24:32.712 INFO:teuthology.orchestra.run.smithi078.stdout:done 2022-03-07T23:24:32.925 INFO:teuthology.orchestra.run.smithi170.stdout:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-07T23:24:32.936 INFO:teuthology.orchestra.run.smithi170.stdout:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-07T23:24:33.081 DEBUG:teuthology.orchestra.run.smithi078:> dpkg -s linux-image-generic-hwe-20.04 2022-03-07T23:24:33.123 INFO:teuthology.orchestra.run.smithi078.stdout:Package: linux-image-generic-hwe-20.04 2022-03-07T23:24:33.123 INFO:teuthology.orchestra.run.smithi078.stdout:Status: install ok installed 2022-03-07T23:24:33.123 INFO:teuthology.orchestra.run.smithi078.stdout:Priority: optional 2022-03-07T23:24:33.124 INFO:teuthology.orchestra.run.smithi078.stdout:Section: kernel 2022-03-07T23:24:33.124 INFO:teuthology.orchestra.run.smithi078.stdout:Installed-Size: 19 2022-03-07T23:24:33.124 INFO:teuthology.orchestra.run.smithi078.stdout:Maintainer: Ubuntu Kernel Team 2022-03-07T23:24:33.124 INFO:teuthology.orchestra.run.smithi078.stdout:Architecture: amd64 2022-03-07T23:24:33.124 INFO:teuthology.orchestra.run.smithi078.stdout:Source: linux-meta-hwe-5.13 2022-03-07T23:24:33.125 INFO:teuthology.orchestra.run.smithi078.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-07T23:24:33.125 INFO:teuthology.orchestra.run.smithi078.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-07T23:24:33.125 INFO:teuthology.orchestra.run.smithi078.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-07T23:24:33.125 INFO:teuthology.orchestra.run.smithi078.stdout:Recommends: thermald 2022-03-07T23:24:33.126 INFO:teuthology.orchestra.run.smithi078.stdout:Description: Generic Linux kernel image 2022-03-07T23:24:33.126 INFO:teuthology.orchestra.run.smithi078.stdout: This package will always depend on the latest generic kernel image 2022-03-07T23:24:33.126 INFO:teuthology.orchestra.run.smithi078.stdout: available. 2022-03-07T23:24:33.126 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-30-generic 2022-03-07T23:24:33.127 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-07T23:24:33.127 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-07T23:24:33.127 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-03-07T23:24:33.128 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi078.front.sepia.ceph.com, path=None, version=distro) 2022-03-07T23:24:33.128 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get clean 2022-03-07T23:24:33.175 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get update 2022-03-07T23:24:33.478 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-07T23:24:33.478 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-07T23:24:33.548 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-07T23:24:33.567 INFO:teuthology.orchestra.run.smithi166.stdout:done 2022-03-07T23:24:33.630 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-07T23:24:33.891 DEBUG:teuthology.orchestra.run.smithi166:> dpkg -s linux-image-generic-hwe-20.04 2022-03-07T23:24:33.921 INFO:teuthology.orchestra.run.smithi166.stdout:Package: linux-image-generic-hwe-20.04 2022-03-07T23:24:33.921 INFO:teuthology.orchestra.run.smithi166.stdout:Status: install ok installed 2022-03-07T23:24:33.921 INFO:teuthology.orchestra.run.smithi166.stdout:Priority: optional 2022-03-07T23:24:33.921 INFO:teuthology.orchestra.run.smithi166.stdout:Section: kernel 2022-03-07T23:24:33.922 INFO:teuthology.orchestra.run.smithi166.stdout:Installed-Size: 19 2022-03-07T23:24:33.922 INFO:teuthology.orchestra.run.smithi166.stdout:Maintainer: Ubuntu Kernel Team 2022-03-07T23:24:33.922 INFO:teuthology.orchestra.run.smithi166.stdout:Architecture: amd64 2022-03-07T23:24:33.922 INFO:teuthology.orchestra.run.smithi166.stdout:Source: linux-meta-hwe-5.13 2022-03-07T23:24:33.923 INFO:teuthology.orchestra.run.smithi166.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-07T23:24:33.923 INFO:teuthology.orchestra.run.smithi166.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-07T23:24:33.923 INFO:teuthology.orchestra.run.smithi166.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-07T23:24:33.923 INFO:teuthology.orchestra.run.smithi166.stdout:Recommends: thermald 2022-03-07T23:24:33.924 INFO:teuthology.orchestra.run.smithi166.stdout:Description: Generic Linux kernel image 2022-03-07T23:24:33.924 INFO:teuthology.orchestra.run.smithi166.stdout: This package will always depend on the latest generic kernel image 2022-03-07T23:24:33.924 INFO:teuthology.orchestra.run.smithi166.stdout: available. 2022-03-07T23:24:33.924 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-30-generic 2022-03-07T23:24:33.925 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-07T23:24:33.925 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-07T23:24:33.925 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2022-03-07T23:24:33.925 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi166.front.sepia.ceph.com, path=None, version=distro) 2022-03-07T23:24:33.926 DEBUG:teuthology.orchestra.run.smithi166:> sudo apt-get clean 2022-03-07T23:24:33.994 DEBUG:teuthology.orchestra.run.smithi166:> sudo apt-get update 2022-03-07T23:24:34.256 INFO:teuthology.orchestra.run.smithi166.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-07T23:24:34.257 INFO:teuthology.orchestra.run.smithi166.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-07T23:24:34.330 INFO:teuthology.orchestra.run.smithi166.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-07T23:24:34.413 INFO:teuthology.orchestra.run.smithi166.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-07T23:24:35.012 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2022-03-07T23:24:35.036 DEBUG:teuthology.orchestra.run.smithi078:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-07T23:24:35.104 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2022-03-07T23:24:35.272 INFO:teuthology.orchestra.run.smithi078.stdout:Building dependency tree... 2022-03-07T23:24:35.272 INFO:teuthology.orchestra.run.smithi078.stdout:Reading state information... 2022-03-07T23:24:35.437 INFO:teuthology.orchestra.run.smithi078.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.30.33~20.04.17). 2022-03-07T23:24:35.438 INFO:teuthology.orchestra.run.smithi078.stdout:0 upgraded, 0 newly installed, 0 to remove and 88 not upgraded. 2022-03-07T23:24:35.439 DEBUG:teuthology.orchestra.run.smithi078:> dpkg -s linux-image-generic-hwe-20.04 2022-03-07T23:24:35.461 INFO:teuthology.orchestra.run.smithi078.stdout:Package: linux-image-generic-hwe-20.04 2022-03-07T23:24:35.461 INFO:teuthology.orchestra.run.smithi078.stdout:Status: install ok installed 2022-03-07T23:24:35.462 INFO:teuthology.orchestra.run.smithi078.stdout:Priority: optional 2022-03-07T23:24:35.462 INFO:teuthology.orchestra.run.smithi078.stdout:Section: kernel 2022-03-07T23:24:35.462 INFO:teuthology.orchestra.run.smithi078.stdout:Installed-Size: 19 2022-03-07T23:24:35.462 INFO:teuthology.orchestra.run.smithi078.stdout:Maintainer: Ubuntu Kernel Team 2022-03-07T23:24:35.462 INFO:teuthology.orchestra.run.smithi078.stdout:Architecture: amd64 2022-03-07T23:24:35.463 INFO:teuthology.orchestra.run.smithi078.stdout:Source: linux-meta-hwe-5.13 2022-03-07T23:24:35.463 INFO:teuthology.orchestra.run.smithi078.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-07T23:24:35.463 INFO:teuthology.orchestra.run.smithi078.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-07T23:24:35.463 INFO:teuthology.orchestra.run.smithi078.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-07T23:24:35.464 INFO:teuthology.orchestra.run.smithi078.stdout:Recommends: thermald 2022-03-07T23:24:35.464 INFO:teuthology.orchestra.run.smithi078.stdout:Description: Generic Linux kernel image 2022-03-07T23:24:35.464 INFO:teuthology.orchestra.run.smithi078.stdout: This package will always depend on the latest generic kernel image 2022-03-07T23:24:35.464 INFO:teuthology.orchestra.run.smithi078.stdout: available. 2022-03-07T23:24:35.465 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2022-03-07T23:24:35.472 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.8A7k3hBANU 2022-03-07T23:24:35.473 DEBUG:teuthology.orchestra.run.smithi078:> sudo cp /boot/grub/grub.cfg /tmp/tmp.8A7k3hBANU 2022-03-07T23:24:35.479 INFO:teuthology.orchestra.run.smithi170.stdout:done 2022-03-07T23:24:35.528 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.8A7k3hBANU 2022-03-07T23:24:35.575 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.8A7k3hBANU is 8KB 2022-03-07T23:24:35.590 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.8A7k3hBANU 2022-03-07T23:24:35.598 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-07T23:24:35.656 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2022-03-07T23:24:35.657 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-07T23:24:35.658 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-07T23:24:35.693 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-30-generic 2022-03-07T23:24:35.694 DEBUG:teuthology.orchestra.run.smithi078:> sudo update-grub 2022-03-07T23:24:35.781 INFO:teuthology.orchestra.run.smithi166.stdout:Reading package lists... 2022-03-07T23:24:35.804 DEBUG:teuthology.orchestra.run.smithi166:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-07T23:24:35.812 DEBUG:teuthology.orchestra.run.smithi170:> dpkg -s linux-image-generic-hwe-20.04 2022-03-07T23:24:35.842 INFO:teuthology.orchestra.run.smithi170.stdout:Package: linux-image-generic-hwe-20.04 2022-03-07T23:24:35.842 INFO:teuthology.orchestra.run.smithi170.stdout:Status: install ok installed 2022-03-07T23:24:35.843 INFO:teuthology.orchestra.run.smithi170.stdout:Priority: optional 2022-03-07T23:24:35.843 INFO:teuthology.orchestra.run.smithi170.stdout:Section: kernel 2022-03-07T23:24:35.843 INFO:teuthology.orchestra.run.smithi170.stdout:Installed-Size: 19 2022-03-07T23:24:35.843 INFO:teuthology.orchestra.run.smithi170.stdout:Maintainer: Ubuntu Kernel Team 2022-03-07T23:24:35.843 INFO:teuthology.orchestra.run.smithi170.stdout:Architecture: amd64 2022-03-07T23:24:35.844 INFO:teuthology.orchestra.run.smithi170.stdout:Source: linux-meta-hwe-5.13 2022-03-07T23:24:35.844 INFO:teuthology.orchestra.run.smithi170.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-07T23:24:35.844 INFO:teuthology.orchestra.run.smithi170.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-07T23:24:35.844 INFO:teuthology.orchestra.run.smithi170.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-07T23:24:35.845 INFO:teuthology.orchestra.run.smithi170.stdout:Recommends: thermald 2022-03-07T23:24:35.845 INFO:teuthology.orchestra.run.smithi170.stdout:Description: Generic Linux kernel image 2022-03-07T23:24:35.845 INFO:teuthology.orchestra.run.smithi170.stdout: This package will always depend on the latest generic kernel image 2022-03-07T23:24:35.845 INFO:teuthology.orchestra.run.smithi170.stdout: available. 2022-03-07T23:24:35.846 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-97-generic Expected: 5.13.0-30-generic 2022-03-07T23:24:35.846 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-03-07T23:24:35.846 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-03-07T23:24:35.847 INFO:teuthology.task.kernel:Installing distro kernel on host.c... 2022-03-07T23:24:35.847 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi170.front.sepia.ceph.com, path=None, version=distro) 2022-03-07T23:24:35.847 DEBUG:teuthology.orchestra.run.smithi170:> sudo apt-get clean 2022-03-07T23:24:35.872 INFO:teuthology.orchestra.run.smithi166.stdout:Reading package lists... 2022-03-07T23:24:35.915 DEBUG:teuthology.orchestra.run.smithi170:> sudo apt-get update 2022-03-07T23:24:36.029 INFO:teuthology.orchestra.run.smithi166.stdout:Building dependency tree... 2022-03-07T23:24:36.030 INFO:teuthology.orchestra.run.smithi166.stdout:Reading state information... 2022-03-07T23:24:36.048 INFO:teuthology.orchestra.run.smithi170.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-07T23:24:36.162 INFO:teuthology.orchestra.run.smithi170.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-07T23:24:36.201 INFO:teuthology.orchestra.run.smithi166.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.30.33~20.04.17). 2022-03-07T23:24:36.201 INFO:teuthology.orchestra.run.smithi166.stdout:0 upgraded, 0 newly installed, 0 to remove and 88 not upgraded. 2022-03-07T23:24:36.202 DEBUG:teuthology.orchestra.run.smithi166:> dpkg -s linux-image-generic-hwe-20.04 2022-03-07T23:24:36.220 INFO:teuthology.orchestra.run.smithi166.stdout:Package: linux-image-generic-hwe-20.04 2022-03-07T23:24:36.220 INFO:teuthology.orchestra.run.smithi166.stdout:Status: install ok installed 2022-03-07T23:24:36.220 INFO:teuthology.orchestra.run.smithi166.stdout:Priority: optional 2022-03-07T23:24:36.221 INFO:teuthology.orchestra.run.smithi166.stdout:Section: kernel 2022-03-07T23:24:36.221 INFO:teuthology.orchestra.run.smithi166.stdout:Installed-Size: 19 2022-03-07T23:24:36.221 INFO:teuthology.orchestra.run.smithi166.stdout:Maintainer: Ubuntu Kernel Team 2022-03-07T23:24:36.221 INFO:teuthology.orchestra.run.smithi166.stdout:Architecture: amd64 2022-03-07T23:24:36.221 INFO:teuthology.orchestra.run.smithi166.stdout:Source: linux-meta-hwe-5.13 2022-03-07T23:24:36.222 INFO:teuthology.orchestra.run.smithi166.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-07T23:24:36.222 INFO:teuthology.orchestra.run.smithi166.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-07T23:24:36.222 INFO:teuthology.orchestra.run.smithi166.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-07T23:24:36.222 INFO:teuthology.orchestra.run.smithi166.stdout:Recommends: thermald 2022-03-07T23:24:36.223 INFO:teuthology.orchestra.run.smithi166.stdout:Description: Generic Linux kernel image 2022-03-07T23:24:36.223 INFO:teuthology.orchestra.run.smithi166.stdout: This package will always depend on the latest generic kernel image 2022-03-07T23:24:36.223 INFO:teuthology.orchestra.run.smithi166.stdout: available. 2022-03-07T23:24:36.224 DEBUG:teuthology.orchestra.run.smithi166:> mktemp 2022-03-07T23:24:36.233 INFO:teuthology.orchestra.run.smithi166.stdout:/tmp/tmp.ahvhCzlO7R 2022-03-07T23:24:36.233 DEBUG:teuthology.orchestra.run.smithi166:> sudo cp /boot/grub/grub.cfg /tmp/tmp.ahvhCzlO7R 2022-03-07T23:24:36.245 INFO:teuthology.orchestra.run.smithi170.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-07T23:24:36.292 DEBUG:teuthology.orchestra.run.smithi166:> sudo chmod 0666 /tmp/tmp.ahvhCzlO7R 2022-03-07T23:24:36.327 INFO:teuthology.orchestra.run.smithi170.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-07T23:24:36.374 DEBUG:teuthology.orchestra.remote:smithi166:/tmp/tmp.ahvhCzlO7R is 8KB 2022-03-07T23:24:36.389 DEBUG:teuthology.orchestra.run.smithi166:> rm -fr /tmp/tmp.ahvhCzlO7R 2022-03-07T23:24:36.395 DEBUG:teuthology.orchestra.run.smithi166:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-07T23:24:36.452 DEBUG:teuthology.orchestra.run.smithi166:> set -ex 2022-03-07T23:24:36.452 DEBUG:teuthology.orchestra.run.smithi166:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-07T23:24:36.453 DEBUG:teuthology.orchestra.run.smithi166:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-07T23:24:36.487 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-30-generic 2022-03-07T23:24:36.488 DEBUG:teuthology.orchestra.run.smithi166:> sudo update-grub 2022-03-07T23:24:36.953 INFO:teuthology.orchestra.run.smithi078.stderr:Sourcing file `/etc/default/grub' 2022-03-07T23:24:36.956 INFO:teuthology.orchestra.run.smithi078.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-07T23:24:36.977 INFO:teuthology.orchestra.run.smithi078.stderr:Generating grub configuration file ... 2022-03-07T23:24:37.212 INFO:teuthology.orchestra.run.smithi078.stderr:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-07T23:24:37.223 INFO:teuthology.orchestra.run.smithi078.stderr:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-07T23:24:37.581 INFO:teuthology.orchestra.run.smithi170.stdout:Reading package lists... 2022-03-07T23:24:37.599 INFO:teuthology.orchestra.run.smithi078.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-07T23:24:37.604 DEBUG:teuthology.orchestra.run.smithi170:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-03-07T23:24:37.610 INFO:teuthology.orchestra.run.smithi078.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-07T23:24:37.656 INFO:teuthology.orchestra.run.smithi166.stderr:Sourcing file `/etc/default/grub' 2022-03-07T23:24:37.658 INFO:teuthology.orchestra.run.smithi166.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-07T23:24:37.669 INFO:teuthology.orchestra.run.smithi170.stdout:Reading package lists... 2022-03-07T23:24:37.674 INFO:teuthology.orchestra.run.smithi166.stderr:Generating grub configuration file ... 2022-03-07T23:24:37.841 INFO:teuthology.orchestra.run.smithi170.stdout:Building dependency tree... 2022-03-07T23:24:37.842 INFO:teuthology.orchestra.run.smithi170.stdout:Reading state information... 2022-03-07T23:24:37.910 INFO:teuthology.orchestra.run.smithi166.stderr:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-07T23:24:37.923 INFO:teuthology.orchestra.run.smithi166.stderr:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-07T23:24:38.018 INFO:teuthology.orchestra.run.smithi170.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.13.0.30.33~20.04.17). 2022-03-07T23:24:38.018 INFO:teuthology.orchestra.run.smithi170.stdout:0 upgraded, 0 newly installed, 0 to remove and 88 not upgraded. 2022-03-07T23:24:38.019 DEBUG:teuthology.orchestra.run.smithi170:> dpkg -s linux-image-generic-hwe-20.04 2022-03-07T23:24:38.039 INFO:teuthology.orchestra.run.smithi170.stdout:Package: linux-image-generic-hwe-20.04 2022-03-07T23:24:38.040 INFO:teuthology.orchestra.run.smithi170.stdout:Status: install ok installed 2022-03-07T23:24:38.040 INFO:teuthology.orchestra.run.smithi170.stdout:Priority: optional 2022-03-07T23:24:38.040 INFO:teuthology.orchestra.run.smithi170.stdout:Section: kernel 2022-03-07T23:24:38.040 INFO:teuthology.orchestra.run.smithi170.stdout:Installed-Size: 19 2022-03-07T23:24:38.041 INFO:teuthology.orchestra.run.smithi170.stdout:Maintainer: Ubuntu Kernel Team 2022-03-07T23:24:38.041 INFO:teuthology.orchestra.run.smithi170.stdout:Architecture: amd64 2022-03-07T23:24:38.041 INFO:teuthology.orchestra.run.smithi170.stdout:Source: linux-meta-hwe-5.13 2022-03-07T23:24:38.041 INFO:teuthology.orchestra.run.smithi170.stdout:Version: 5.13.0.30.33~20.04.17 2022-03-07T23:24:38.042 INFO:teuthology.orchestra.run.smithi170.stdout:Provides: virtualbox-guest-modules (= 5.13.0-30), wireguard-modules (= 1.0.0), zfs-modules (= 2.0.6-1ubuntu2) 2022-03-07T23:24:38.042 INFO:teuthology.orchestra.run.smithi170.stdout:Depends: linux-image-5.13.0-30-generic, linux-modules-extra-5.13.0-30-generic, linux-firmware, intel-microcode, amd64-microcode 2022-03-07T23:24:38.043 INFO:teuthology.orchestra.run.smithi170.stdout:Recommends: thermald 2022-03-07T23:24:38.043 INFO:teuthology.orchestra.run.smithi170.stdout:Description: Generic Linux kernel image 2022-03-07T23:24:38.043 INFO:teuthology.orchestra.run.smithi170.stdout: This package will always depend on the latest generic kernel image 2022-03-07T23:24:38.043 INFO:teuthology.orchestra.run.smithi170.stdout: available. 2022-03-07T23:24:38.044 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2022-03-07T23:24:38.049 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.cnQEvaXYYu 2022-03-07T23:24:38.049 DEBUG:teuthology.orchestra.run.smithi170:> sudo cp /boot/grub/grub.cfg /tmp/tmp.cnQEvaXYYu 2022-03-07T23:24:38.107 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.cnQEvaXYYu 2022-03-07T23:24:38.147 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.cnQEvaXYYu is 8KB 2022-03-07T23:24:38.161 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.cnQEvaXYYu 2022-03-07T23:24:38.168 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-03-07T23:24:38.226 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2022-03-07T23:24:38.227 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-03-07T23:24:38.227 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-03-07T23:24:38.260 INFO:teuthology.orchestra.run.smithi166.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-07T23:24:38.380 INFO:teuthology.orchestra.run.smithi166.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-07T23:24:38.381 INFO:teuthology.task.kernel:Distro Kernel Version: 5.13.0-30-generic 2022-03-07T23:24:38.382 DEBUG:teuthology.orchestra.run.smithi170:> sudo update-grub 2022-03-07T23:24:39.174 INFO:teuthology.orchestra.run.smithi078.stderr:done 2022-03-07T23:24:39.176 DEBUG:teuthology.orchestra.run.smithi078:> sudo shutdown -r now 2022-03-07T23:24:39.546 INFO:teuthology.orchestra.run.smithi170.stderr:Sourcing file `/etc/default/grub' 2022-03-07T23:24:39.551 INFO:teuthology.orchestra.run.smithi170.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-03-07T23:24:39.569 INFO:teuthology.orchestra.run.smithi170.stderr:Generating grub configuration file ... 2022-03-07T23:24:39.855 INFO:teuthology.orchestra.run.smithi166.stderr:done 2022-03-07T23:24:39.857 DEBUG:teuthology.orchestra.run.smithi166:> sudo shutdown -r now 2022-03-07T23:24:39.892 INFO:teuthology.orchestra.run.smithi170.stderr:Found linux image: /boot/vmlinuz-5.13.0-30-generic 2022-03-07T23:24:39.904 INFO:teuthology.orchestra.run.smithi170.stderr:Found initrd image: /boot/initrd.img-5.13.0-30-generic 2022-03-07T23:24:40.316 INFO:teuthology.orchestra.run.smithi170.stderr:Found linux image: /boot/vmlinuz-5.4.0-97-generic 2022-03-07T23:24:40.327 INFO:teuthology.orchestra.run.smithi170.stderr:Found initrd image: /boot/initrd.img-5.4.0-97-generic 2022-03-07T23:24:41.933 INFO:teuthology.orchestra.run.smithi170.stderr:done 2022-03-07T23:24:41.935 DEBUG:teuthology.orchestra.run.smithi170:> sudo shutdown -r now 2022-03-07T23:25:09.206 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-03-07T23:25:09.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2022-03-07T23:25:09.208 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T23:25:09.861 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2022-03-07T23:25:09.862 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi166.front.sepia.ceph.com' 2022-03-07T23:25:09.862 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi166.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T23:25:11.940 INFO:teuthology.task.kernel:Checking client host.c for new kernel version... 2022-03-07T23:25:11.940 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2022-03-07T23:25:11.941 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T23:26:09.253 DEBUG:teuthology.orchestra.remote:timed out 2022-03-07T23:26:09.865 DEBUG:teuthology.orchestra.remote:timed out 2022-03-07T23:26:11.945 DEBUG:teuthology.orchestra.remote:timed out 2022-03-07T23:26:39.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2022-03-07T23:26:39.284 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T23:26:39.479 DEBUG:teuthology.orchestra.run.smithi078:> true 2022-03-07T23:26:39.866 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi166.front.sepia.ceph.com' 2022-03-07T23:26:39.867 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi166.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T23:26:40.011 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2022-03-07T23:26:40.012 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.13.0-30-generic"... 2022-03-07T23:26:40.012 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2022-03-07T23:26:40.019 INFO:teuthology.orchestra.run.smithi078.stdout:5.13.0-30-generic 2022-03-07T23:26:40.019 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-30-generic vs 5.13.0-30-generic 2022-03-07T23:26:40.020 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-07T23:26:40.020 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-07T23:26:40.051 DEBUG:teuthology.orchestra.run.smithi166:> true 2022-03-07T23:26:40.552 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi166.front.sepia.ceph.com' 2022-03-07T23:26:40.552 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.13.0-30-generic"... 2022-03-07T23:26:40.553 DEBUG:teuthology.orchestra.run.smithi166:> uname -r 2022-03-07T23:26:40.558 INFO:teuthology.orchestra.run.smithi166.stdout:5.13.0-30-generic 2022-03-07T23:26:40.558 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-30-generic vs 5.13.0-30-generic 2022-03-07T23:26:40.559 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-07T23:26:40.559 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-07T23:26:41.021 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-03-07T23:26:41.021 DEBUG:teuthology.orchestra.run.smithi078:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-07T23:26:41.378 INFO:teuthology.orchestra.run.smithi078.stdout:ttyS1 2022-03-07T23:26:41.405 DEBUG:teuthology.parallel:result is None 2022-03-07T23:26:41.560 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2022-03-07T23:26:41.560 DEBUG:teuthology.orchestra.run.smithi166:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-07T23:26:41.629 INFO:teuthology.orchestra.run.smithi166.stdout:ttyS1 2022-03-07T23:26:41.666 DEBUG:teuthology.parallel:result is None 2022-03-07T23:26:41.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2022-03-07T23:26:41.948 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T23:26:42.122 DEBUG:teuthology.orchestra.run.smithi170:> true 2022-03-07T23:26:42.722 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2022-03-07T23:26:42.723 INFO:teuthology.task.kernel:Checking kernel version of host.c, want "5.13.0-30-generic"... 2022-03-07T23:26:42.723 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2022-03-07T23:26:42.728 INFO:teuthology.orchestra.run.smithi170.stdout:5.13.0-30-generic 2022-03-07T23:26:42.728 DEBUG:teuthology.task.kernel:current kernel version is 5.13.0-30-generic vs 5.13.0-30-generic 2022-03-07T23:26:42.728 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-03-07T23:26:42.729 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-03-07T23:26:43.729 INFO:teuthology.task.kernel:Enabling kdb on host.c... 2022-03-07T23:26:43.730 DEBUG:teuthology.orchestra.run.smithi170:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-03-07T23:26:43.797 INFO:teuthology.orchestra.run.smithi170.stdout:ttyS1 2022-03-07T23:26:43.817 DEBUG:teuthology.parallel:result is None 2022-03-07T23:26:43.818 INFO:teuthology.run_tasks:Running task internal.base... 2022-03-07T23:26:43.831 INFO:teuthology.task.internal:Creating test directory... 2022-03-07T23:26:43.831 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-07T23:26:43.835 DEBUG:teuthology.orchestra.run.smithi166:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-07T23:26:43.840 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-03-07T23:26:43.847 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-03-07T23:26:43.879 INFO:teuthology.run_tasks:Running task internal.archive... 2022-03-07T23:26:43.912 INFO:teuthology.task.internal:Creating archive directory... 2022-03-07T23:26:43.912 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-07T23:26:43.915 DEBUG:teuthology.orchestra.run.smithi166:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-07T23:26:43.918 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-03-07T23:26:44.181 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-03-07T23:26:44.192 INFO:teuthology.task.internal:Enabling coredump saving... 2022-03-07T23:26:44.193 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-07T23:26:44.196 DEBUG:teuthology.orchestra.run.smithi166:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-07T23:26:44.198 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-03-07T23:26:44.220 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-07T23:26:44.222 INFO:teuthology.orchestra.run.smithi166.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-07T23:26:44.228 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-07T23:26:44.229 INFO:teuthology.orchestra.run.smithi166.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-07T23:26:44.232 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-07T23:26:44.238 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-03-07T23:26:44.240 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-03-07T23:26:44.252 INFO:teuthology.task.internal:Configuring sudo... 2022-03-07T23:26:44.253 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-07T23:26:44.276 DEBUG:teuthology.orchestra.run.smithi166:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-07T23:26:44.279 DEBUG:teuthology.orchestra.run.smithi170:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-03-07T23:26:44.300 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-03-07T23:26:44.310 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-03-07T23:26:44.311 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-07T23:26:44.314 DEBUG:teuthology.orchestra.run.smithi166:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-07T23:26:44.337 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-03-07T23:26:44.346 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-07T23:26:44.369 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-07T23:26:44.375 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2022-03-07T23:26:44.376 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-07T23:26:44.401 DEBUG:teuthology.orchestra.run.smithi166:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-07T23:26:44.409 DEBUG:teuthology.orchestra.run.smithi166:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-07T23:26:44.458 DEBUG:teuthology.orchestra.run.smithi166:> set -ex 2022-03-07T23:26:44.458 DEBUG:teuthology.orchestra.run.smithi166:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-07T23:26:44.484 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-03-07T23:26:44.492 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-03-07T23:26:44.543 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2022-03-07T23:26:44.543 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-03-07T23:26:44.574 DEBUG:teuthology.orchestra.run.smithi078:> sudo service rsyslog restart 2022-03-07T23:26:44.577 DEBUG:teuthology.orchestra.run.smithi166:> sudo service rsyslog restart 2022-03-07T23:26:44.580 DEBUG:teuthology.orchestra.run.smithi170:> sudo service rsyslog restart 2022-03-07T23:26:44.632 INFO:teuthology.run_tasks:Running task internal.timer... 2022-03-07T23:26:44.644 INFO:teuthology.task.internal:Starting timer... 2022-03-07T23:26:44.644 INFO:teuthology.run_tasks:Running task pcp... 2022-03-07T23:26:44.698 INFO:teuthology.run_tasks:Running task selinux... 2022-03-07T23:26:44.729 DEBUG:teuthology.task.selinux:Excluding smithi078: OS 'ubuntu' does not support SELinux 2022-03-07T23:26:44.730 DEBUG:teuthology.task.selinux:Excluding smithi166: OS 'ubuntu' does not support SELinux 2022-03-07T23:26:44.730 DEBUG:teuthology.task.selinux:Excluding smithi170: OS 'ubuntu' does not support SELinux 2022-03-07T23:26:44.731 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-03-07T23:26:44.731 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-03-07T23:26:44.731 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-03-07T23:26:44.732 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-03-07T23:26:44.764 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2022-03-07T23:26:44.765 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-03-07T23:26:44.789 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-03-07T23:26:44.791 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi078.front.sepia.ceph.com,smithi166.front.sepia.ceph.com,smithi170.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-03-07T23:29:53.986 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi078.front.sepia.ceph.com'), Remote(name='ubuntu@smithi166.front.sepia.ceph.com'), Remote(name='ubuntu@smithi170.front.sepia.ceph.com')] 2022-03-07T23:29:53.989 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2022-03-07T23:29:53.990 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T23:29:54.041 DEBUG:teuthology.orchestra.run.smithi078:> true 2022-03-07T23:29:54.165 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi166.front.sepia.ceph.com' 2022-03-07T23:29:54.166 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi166.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T23:29:54.216 DEBUG:teuthology.orchestra.run.smithi166:> true 2022-03-07T23:29:54.337 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2022-03-07T23:29:54.337 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-03-07T23:29:54.380 DEBUG:teuthology.orchestra.run.smithi170:> true 2022-03-07T23:29:54.496 INFO:teuthology.run_tasks:Running task clock... 2022-03-07T23:29:54.513 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-03-07T23:29:54.513 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-07T23:29:54.515 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-07T23:29:54.518 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-07T23:29:54.519 DEBUG:teuthology.orchestra.run.smithi166:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-07T23:29:54.522 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-03-07T23:29:54.522 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-03-07T23:29:54.550 INFO:teuthology.orchestra.run.smithi078.stdout: 7 Mar 23:29:54 ntpd[9005]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-07T23:29:54.550 INFO:teuthology.orchestra.run.smithi078.stdout: 7 Mar 23:29:54 ntpd[9005]: Command line: ntpd -gq 2022-03-07T23:29:54.551 INFO:teuthology.orchestra.run.smithi078.stdout: 7 Mar 23:29:54 ntpd[9005]: proto: precision = 0.074 usec (-24) 2022-03-07T23:29:54.552 INFO:teuthology.orchestra.run.smithi078.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-07T23:29:54.552 INFO:teuthology.orchestra.run.smithi078.stdout: 7 Mar 23:29:54 ntpd[9005]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-07T23:29:54.552 INFO:teuthology.orchestra.run.smithi078.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-07T23:29:54.554 INFO:teuthology.orchestra.run.smithi078.stdout: 7 Mar 23:29:54 ntpd[9005]: restrict ::: KOD does nothing without LIMITED. 2022-03-07T23:29:54.554 INFO:teuthology.orchestra.run.smithi078.stdout: 7 Mar 23:29:54 ntpd[9005]: Listen and drop on 0 v6wildcard [::]:123 2022-03-07T23:29:54.554 INFO:teuthology.orchestra.run.smithi078.stdout: 7 Mar 23:29:54 ntpd[9005]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-07T23:29:54.554 INFO:teuthology.orchestra.run.smithi078.stdout: 7 Mar 23:29:54 ntpd[9005]: Listen normally on 2 lo 127.0.0.1:123 2022-03-07T23:29:54.555 INFO:teuthology.orchestra.run.smithi078.stdout: 7 Mar 23:29:54 ntpd[9005]: Listen normally on 3 enp3s0f1 172.21.15.78:123 2022-03-07T23:29:54.555 INFO:teuthology.orchestra.run.smithi078.stdout: 7 Mar 23:29:54 ntpd[9005]: Listen normally on 4 lo [::1]:123 2022-03-07T23:29:54.555 INFO:teuthology.orchestra.run.smithi078.stdout: 7 Mar 23:29:54 ntpd[9005]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d4dd%5]:123 2022-03-07T23:29:54.555 INFO:teuthology.orchestra.run.smithi078.stdout: 7 Mar 23:29:54 ntpd[9005]: Listening on routing socket on fd #22 for interface updates 2022-03-07T23:29:54.556 INFO:teuthology.orchestra.run.smithi170.stdout: 7 Mar 23:29:54 ntpd[9064]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-07T23:29:54.557 INFO:teuthology.orchestra.run.smithi170.stdout: 7 Mar 23:29:54 ntpd[9064]: Command line: ntpd -gq 2022-03-07T23:29:54.557 INFO:teuthology.orchestra.run.smithi170.stdout: 7 Mar 23:29:54 ntpd[9064]: proto: precision = 0.074 usec (-24) 2022-03-07T23:29:54.558 INFO:teuthology.orchestra.run.smithi170.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-07T23:29:54.558 INFO:teuthology.orchestra.run.smithi166.stdout: 7 Mar 23:29:54 ntpd[9085]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-03-07T23:29:54.558 INFO:teuthology.orchestra.run.smithi166.stdout: 7 Mar 23:29:54 ntpd[9085]: Command line: ntpd -gq 2022-03-07T23:29:54.558 INFO:teuthology.orchestra.run.smithi166.stdout: 7 Mar 23:29:54 ntpd[9085]: proto: precision = 0.070 usec (-24) 2022-03-07T23:29:54.559 INFO:teuthology.orchestra.run.smithi166.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-07T23:29:54.559 INFO:teuthology.orchestra.run.smithi170.stdout: 7 Mar 23:29:54 ntpd[9064]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-07T23:29:54.560 INFO:teuthology.orchestra.run.smithi170.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-07T23:29:54.560 INFO:teuthology.orchestra.run.smithi166.stdout: 7 Mar 23:29:54 ntpd[9085]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-03-07T23:29:54.561 INFO:teuthology.orchestra.run.smithi166.stderr:restrict ::: KOD does nothing without LIMITED. 2022-03-07T23:29:54.563 INFO:teuthology.orchestra.run.smithi170.stdout: 7 Mar 23:29:54 ntpd[9064]: restrict ::: KOD does nothing without LIMITED. 2022-03-07T23:29:54.563 INFO:teuthology.orchestra.run.smithi170.stdout: 7 Mar 23:29:54 ntpd[9064]: Listen and drop on 0 v6wildcard [::]:123 2022-03-07T23:29:54.563 INFO:teuthology.orchestra.run.smithi170.stdout: 7 Mar 23:29:54 ntpd[9064]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-07T23:29:54.563 INFO:teuthology.orchestra.run.smithi170.stdout: 7 Mar 23:29:54 ntpd[9064]: Listen normally on 2 lo 127.0.0.1:123 2022-03-07T23:29:54.564 INFO:teuthology.orchestra.run.smithi170.stdout: 7 Mar 23:29:54 ntpd[9064]: Listen normally on 3 enp3s0f1 172.21.15.170:123 2022-03-07T23:29:54.564 INFO:teuthology.orchestra.run.smithi170.stdout: 7 Mar 23:29:54 ntpd[9064]: Listen normally on 4 lo [::1]:123 2022-03-07T23:29:54.564 INFO:teuthology.orchestra.run.smithi170.stdout: 7 Mar 23:29:54 ntpd[9064]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7255%5]:123 2022-03-07T23:29:54.565 INFO:teuthology.orchestra.run.smithi170.stdout: 7 Mar 23:29:54 ntpd[9064]: Listening on routing socket on fd #22 for interface updates 2022-03-07T23:29:54.565 INFO:teuthology.orchestra.run.smithi166.stdout: 7 Mar 23:29:54 ntpd[9085]: restrict ::: KOD does nothing without LIMITED. 2022-03-07T23:29:54.565 INFO:teuthology.orchestra.run.smithi166.stdout: 7 Mar 23:29:54 ntpd[9085]: Listen and drop on 0 v6wildcard [::]:123 2022-03-07T23:29:54.566 INFO:teuthology.orchestra.run.smithi166.stdout: 7 Mar 23:29:54 ntpd[9085]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-03-07T23:29:54.566 INFO:teuthology.orchestra.run.smithi166.stdout: 7 Mar 23:29:54 ntpd[9085]: Listen normally on 2 lo 127.0.0.1:123 2022-03-07T23:29:54.566 INFO:teuthology.orchestra.run.smithi166.stdout: 7 Mar 23:29:54 ntpd[9085]: Listen normally on 3 enp3s0f1 172.21.15.166:123 2022-03-07T23:29:54.566 INFO:teuthology.orchestra.run.smithi166.stdout: 7 Mar 23:29:54 ntpd[9085]: Listen normally on 4 lo [::1]:123 2022-03-07T23:29:54.567 INFO:teuthology.orchestra.run.smithi166.stdout: 7 Mar 23:29:54 ntpd[9085]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:77ed%5]:123 2022-03-07T23:29:54.567 INFO:teuthology.orchestra.run.smithi166.stdout: 7 Mar 23:29:54 ntpd[9085]: Listening on routing socket on fd #22 for interface updates 2022-03-07T23:29:55.552 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:55 ntpd[9005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:55.552 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:55 ntpd[9005]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:55.553 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:55 ntpd[9005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:55.553 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:55 ntpd[9005]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:55.553 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:55 ntpd[9005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:55.553 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:55 ntpd[9005]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:55.554 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:55 ntpd[9005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:55.554 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:55 ntpd[9005]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:55.554 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:55 ntpd[9005]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:55.554 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:55 ntpd[9005]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:55.555 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:55 ntpd[9005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:55.555 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:55 ntpd[9005]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:55.555 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:55 ntpd[9005]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:55.555 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:55 ntpd[9005]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:55.556 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:55 ntpd[9005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:55.556 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:55 ntpd[9005]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:55.556 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:55 ntpd[9005]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:55.556 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:55 ntpd[9005]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:55.558 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:55 ntpd[9064]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:55.558 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:55 ntpd[9064]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:55.559 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:55 ntpd[9064]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:55.559 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:55 ntpd[9064]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:55.559 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:55 ntpd[9064]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:55.559 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:55 ntpd[9064]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:55.560 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:55 ntpd[9085]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:55.561 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:55 ntpd[9085]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:55.561 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:55 ntpd[9085]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:55.561 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:55 ntpd[9085]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:55.561 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:55 ntpd[9085]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:55.562 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:55 ntpd[9085]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:55.562 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:55 ntpd[9085]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:55.562 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:55 ntpd[9085]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:55.562 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:55 ntpd[9085]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:55.563 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:55 ntpd[9085]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:55.563 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:55 ntpd[9085]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:55.563 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:55 ntpd[9085]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:56.553 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:56 ntpd[9085]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:56.553 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:56 ntpd[9085]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:56.554 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:56 ntpd[9085]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:56.554 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:56 ntpd[9085]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:56.554 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:56 ntpd[9085]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:56.554 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:56 ntpd[9085]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:56.555 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:56 ntpd[9064]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:56.556 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:56 ntpd[9064]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:56.556 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:56 ntpd[9064]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:56.556 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:56 ntpd[9064]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:56.556 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:56 ntpd[9064]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:56.556 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:56 ntpd[9064]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:57.553 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:57 ntpd[9005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:57.554 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:57 ntpd[9005]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:57.554 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:57 ntpd[9005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:57.554 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:57 ntpd[9005]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:57.555 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:57 ntpd[9005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:57.555 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:57 ntpd[9005]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:57.555 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:57 ntpd[9005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:57.555 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:57 ntpd[9005]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:57.555 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:57 ntpd[9005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:57.556 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:57 ntpd[9005]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:57.556 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:57 ntpd[9005]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:57.556 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:57 ntpd[9005]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:57.556 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:57 ntpd[9005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:57.557 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:57 ntpd[9005]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:57.557 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:57 ntpd[9005]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:57.557 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:57 ntpd[9005]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:57.558 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:57 ntpd[9005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:57.558 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:57 ntpd[9005]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:57.558 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:57 ntpd[9005]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:57.558 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:57 ntpd[9005]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:57.559 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:57 ntpd[9005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:57.559 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:57 ntpd[9005]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:57.559 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:57 ntpd[9005]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:57.559 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:57 ntpd[9005]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:57.562 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:57 ntpd[9064]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:57.563 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:57 ntpd[9064]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:57.563 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:57 ntpd[9064]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:57.563 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:57 ntpd[9064]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:57.563 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:57 ntpd[9064]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:57.564 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:57 ntpd[9064]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:57.564 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:57 ntpd[9064]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:57.564 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:57 ntpd[9064]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:57.564 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:57 ntpd[9064]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:57.564 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:57 ntpd[9064]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:57.565 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:57 ntpd[9064]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:57.565 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:57 ntpd[9064]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:57.566 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:57 ntpd[9085]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:57.566 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:57 ntpd[9085]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:57.566 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:57 ntpd[9085]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:57.567 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:57 ntpd[9085]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:57.567 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:57 ntpd[9085]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:57.567 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:57 ntpd[9085]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:57.568 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:57 ntpd[9085]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:57.568 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:57 ntpd[9085]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:57.568 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:57 ntpd[9085]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:57.569 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:57 ntpd[9085]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:57.569 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:57 ntpd[9085]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:57.569 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:57 ntpd[9085]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:57.569 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:57 ntpd[9085]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:57.569 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:57 ntpd[9085]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:57.570 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:57 ntpd[9085]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:57.570 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:57 ntpd[9085]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:57.570 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:57 ntpd[9085]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:57.570 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:57 ntpd[9085]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:58.553 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:58 ntpd[9085]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:58.557 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:58 ntpd[9085]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:58.558 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:58 ntpd[9085]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:58.558 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:58 ntpd[9085]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:58.559 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:58 ntpd[9085]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:58.559 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:58 ntpd[9085]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:58.561 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:58 ntpd[9064]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:58.561 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:58 ntpd[9064]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:58.561 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:58 ntpd[9064]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:58.561 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:58 ntpd[9064]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:58.562 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:58 ntpd[9064]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:58.562 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:58 ntpd[9064]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:58.562 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:58 ntpd[9064]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:58.563 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:58 ntpd[9064]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:58.563 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:58 ntpd[9064]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:58.563 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:58 ntpd[9064]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:58.564 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:58 ntpd[9064]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:58.564 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:58 ntpd[9064]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:59.552 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:59 ntpd[9005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:59.553 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:59 ntpd[9005]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:59.553 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:59 ntpd[9005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:59.553 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:59 ntpd[9005]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:59.554 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:59 ntpd[9005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:59.554 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:59 ntpd[9005]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:59.554 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:59 ntpd[9005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:59.554 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:59 ntpd[9005]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:59.555 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:59 ntpd[9005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:59.555 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:59 ntpd[9005]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:59.555 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:59 ntpd[9005]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:59.555 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:59 ntpd[9005]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:59.555 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:59 ntpd[9005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:59.556 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:59 ntpd[9005]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:59.556 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:59 ntpd[9005]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:59.556 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:59 ntpd[9005]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:59.557 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:59 ntpd[9005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:59.557 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:59 ntpd[9005]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:59.557 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:59 ntpd[9005]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:59.558 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:59 ntpd[9005]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:59.558 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:59 ntpd[9005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:59.558 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:59 ntpd[9005]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:59.558 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:59 ntpd[9005]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:59.558 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:29:59 ntpd[9005]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:59.562 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:59 ntpd[9064]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:59.562 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:59 ntpd[9064]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:59.562 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:59 ntpd[9064]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:59.563 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:59 ntpd[9064]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:59.563 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:59 ntpd[9064]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:59.563 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:59 ntpd[9064]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:59.563 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:59 ntpd[9064]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:59.564 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:59 ntpd[9064]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:59.564 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:59 ntpd[9064]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:59.564 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:59 ntpd[9064]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:59.564 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:59 ntpd[9064]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:59.565 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:29:59 ntpd[9064]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:59.566 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:59 ntpd[9085]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:59.566 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:59 ntpd[9085]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:59.566 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:59 ntpd[9085]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:59.567 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:59 ntpd[9085]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:59.567 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:59 ntpd[9085]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:59.567 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:59 ntpd[9085]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:59.567 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:59 ntpd[9085]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:59.568 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:59 ntpd[9085]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:59.568 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:59 ntpd[9085]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:59.568 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:59 ntpd[9085]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:59.568 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:59 ntpd[9085]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:59.568 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:59 ntpd[9085]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:59.569 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:59 ntpd[9085]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:59.569 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:59 ntpd[9085]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:29:59.569 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:59 ntpd[9085]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:29:59.569 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:59 ntpd[9085]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:29:59.570 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:59 ntpd[9085]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:29:59.570 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:29:59 ntpd[9085]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:30:00.553 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:30:00 ntpd[9085]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:30:00.553 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:30:00 ntpd[9085]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:30:00.553 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:30:00 ntpd[9085]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:30:00.554 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:30:00 ntpd[9085]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:30:00.554 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:30:00 ntpd[9085]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:30:00.554 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:30:00 ntpd[9085]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:30:00.556 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:30:00 ntpd[9064]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:30:00.556 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:30:00 ntpd[9064]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:30:00.556 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:30:00 ntpd[9064]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:30:00.557 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:30:00 ntpd[9064]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:30:00.557 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:30:00 ntpd[9064]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:30:00.557 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:30:00 ntpd[9064]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:30:00.557 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:30:00 ntpd[9064]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:30:00.558 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:30:00 ntpd[9064]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:30:00.558 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:30:00 ntpd[9064]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:30:00.558 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:30:00 ntpd[9064]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:30:00.558 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:30:00 ntpd[9064]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:30:00.558 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:30:00 ntpd[9064]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:30:01.551 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:30:01 ntpd[9005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:30:01.552 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:30:01 ntpd[9005]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:30:01.552 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:30:01 ntpd[9005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:30:01.552 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:30:01 ntpd[9005]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:30:01.552 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:30:01 ntpd[9005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:30:01.553 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:30:01 ntpd[9005]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:30:01.553 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:30:01 ntpd[9005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:30:01.553 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:30:01 ntpd[9005]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:30:01.553 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:30:01 ntpd[9005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:30:01.553 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:30:01 ntpd[9005]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:30:01.554 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:30:01 ntpd[9005]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:30:01.554 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:30:01 ntpd[9005]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:30:01.554 INFO:teuthology.orchestra.run.smithi078.stdout: 7 Mar 23:30:01 ntpd[9005]: ntpd: time slew -0.000649 s 2022-03-07T23:30:01.555 INFO:teuthology.orchestra.run.smithi078.stdout:ntpd: time slew -0.000649s 2022-03-07T23:30:01.555 INFO:teuthology.orchestra.run.smithi078.stderr: 7 Mar 23:30:01 ntpd[9005]: can't open /var/log/ntpstats/loopstats.20220307: Permission denied 2022-03-07T23:30:01.557 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:30:01 ntpd[9064]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:30:01.557 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:30:01 ntpd[9064]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:30:01.558 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:30:01 ntpd[9064]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:30:01.558 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:30:01 ntpd[9064]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:30:01.558 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:30:01 ntpd[9064]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:30:01.558 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:30:01 ntpd[9064]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:30:01.559 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:30:01 ntpd[9064]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:30:01.559 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:30:01 ntpd[9064]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:30:01.559 INFO:teuthology.orchestra.run.smithi170.stdout: 7 Mar 23:30:01 ntpd[9064]: ntpd: time slew -0.000961 s 2022-03-07T23:30:01.560 INFO:teuthology.orchestra.run.smithi170.stdout:ntpd: time slew -0.000961s 2022-03-07T23:30:01.560 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:30:01 ntpd[9085]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:30:01.561 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:30:01 ntpd[9085]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:30:01.561 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:30:01 ntpd[9085]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:30:01.561 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:30:01 ntpd[9085]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:30:01.561 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:30:01 ntpd[9085]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:30:01.562 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:30:01 ntpd[9085]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:30:01.562 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:30:01 ntpd[9085]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-03-07T23:30:01.562 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:30:01 ntpd[9085]: can't open /var/log/ntpstats/rawstats.20220307: Permission denied 2022-03-07T23:30:01.562 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:30:01 ntpd[9085]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-03-07T23:30:01.563 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:30:01 ntpd[9085]: can't open /var/log/ntpstats/peerstats.20220307: Permission denied 2022-03-07T23:30:01.563 INFO:teuthology.orchestra.run.smithi170.stderr: 7 Mar 23:30:01 ntpd[9064]: can't open /var/log/ntpstats/loopstats.20220307: Permission denied 2022-03-07T23:30:01.563 INFO:teuthology.orchestra.run.smithi166.stdout: 7 Mar 23:30:01 ntpd[9085]: ntpd: time slew -0.000580 s 2022-03-07T23:30:01.564 INFO:teuthology.orchestra.run.smithi166.stdout:ntpd: time slew -0.000580s 2022-03-07T23:30:01.564 INFO:teuthology.orchestra.run.smithi166.stderr: 7 Mar 23:30:01 ntpd[9085]: can't open /var/log/ntpstats/loopstats.20220307: Permission denied 2022-03-07T23:30:01.611 INFO:teuthology.orchestra.run.smithi166.stdout: remote refid st t when poll reach delay offset jitter 2022-03-07T23:30:01.611 INFO:teuthology.orchestra.run.smithi166.stdout:============================================================================== 2022-03-07T23:30:01.611 INFO:teuthology.orchestra.run.smithi166.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-07T23:30:01.612 INFO:teuthology.orchestra.run.smithi166.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-07T23:30:01.612 INFO:teuthology.orchestra.run.smithi166.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-07T23:30:01.612 INFO:teuthology.orchestra.run.smithi166.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-07T23:30:01.614 INFO:teuthology.orchestra.run.smithi078.stdout: remote refid st t when poll reach delay offset jitter 2022-03-07T23:30:01.614 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================== 2022-03-07T23:30:01.615 INFO:teuthology.orchestra.run.smithi078.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-07T23:30:01.615 INFO:teuthology.orchestra.run.smithi078.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-07T23:30:01.615 INFO:teuthology.orchestra.run.smithi078.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-07T23:30:01.615 INFO:teuthology.orchestra.run.smithi078.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-07T23:30:01.617 INFO:teuthology.orchestra.run.smithi170.stdout: remote refid st t when poll reach delay offset jitter 2022-03-07T23:30:01.617 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================== 2022-03-07T23:30:01.618 INFO:teuthology.orchestra.run.smithi170.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-07T23:30:01.618 INFO:teuthology.orchestra.run.smithi170.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-07T23:30:01.618 INFO:teuthology.orchestra.run.smithi170.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-07T23:30:01.618 INFO:teuthology.orchestra.run.smithi170.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-03-07T23:30:01.619 INFO:teuthology.run_tasks:Running task kubeadm... 2022-03-07T23:30:01.635 INFO:tasks.kubeadm:Kubeadm start 2022-03-07T23:30:01.636 INFO:tasks.kubeadm:Config: {'pod_network': 'flannel', 'version': '1.21'} 2022-03-07T23:30:01.636 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-07T23:30:01.639 DEBUG:teuthology.orchestra.run.smithi166:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-07T23:30:01.657 DEBUG:teuthology.orchestra.run.smithi170:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-03-07T23:30:01.660 INFO:teuthology.orchestra.run.smithi078.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-07T23:30:01.669 INFO:teuthology.orchestra.run.smithi078.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-07T23:30:01.678 INFO:teuthology.orchestra.run.smithi078.stdout:net.ipv4.ip_forward = 1 2022-03-07T23:30:01.681 INFO:teuthology.orchestra.run.smithi166.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-07T23:30:01.685 INFO:teuthology.orchestra.run.smithi170.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-03-07T23:30:01.692 INFO:teuthology.orchestra.run.smithi166.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-07T23:30:01.694 INFO:teuthology.orchestra.run.smithi170.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-03-07T23:30:01.700 INFO:teuthology.orchestra.run.smithi166.stdout:net.ipv4.ip_forward = 1 2022-03-07T23:30:01.701 INFO:teuthology.orchestra.run.smithi170.stdout:net.ipv4.ip_forward = 1 2022-03-07T23:30:01.746 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2022-03-07T23:30:01.746 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-07T23:30:01.783 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2022-03-07T23:30:01.783 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/docker/daemon.json 2022-03-07T23:30:01.795 DEBUG:teuthology.orchestra.run.smithi166:> set -ex 2022-03-07T23:30:01.796 DEBUG:teuthology.orchestra.run.smithi166:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-07T23:30:01.811 DEBUG:teuthology.orchestra.run.smithi166:> set -ex 2022-03-07T23:30:01.812 DEBUG:teuthology.orchestra.run.smithi166:> sudo dd of=/etc/docker/daemon.json 2022-03-07T23:30:01.871 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2022-03-07T23:30:01.871 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-03-07T23:30:01.886 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2022-03-07T23:30:01.886 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/docker/daemon.json 2022-03-07T23:30:01.937 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl restart docker || true 2022-03-07T23:30:01.939 DEBUG:teuthology.orchestra.run.smithi166:> sudo systemctl restart docker || true 2022-03-07T23:30:01.942 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl restart docker || true 2022-03-07T23:30:02.949 INFO:tasks.kubeadm:Identifying pod subnet 2022-03-07T23:30:02.955 INFO:tasks.kubeadm:172.21.15.78 is in 172.21.0.0/20 at pos 3917 2022-03-07T23:30:02.970 INFO:tasks.kubeadm:Pod subnet: 10.250.104.0/21 2022-03-07T23:30:02.970 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-03-07T23:30:02.971 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-07T23:30:02.974 DEBUG:teuthology.orchestra.run.smithi166:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-07T23:30:02.976 DEBUG:teuthology.orchestra.run.smithi170:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-03-07T23:30:02.999 INFO:teuthology.orchestra.run.smithi166.stderr: 2022-03-07T23:30:03.000 INFO:teuthology.orchestra.run.smithi166.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-07T23:30:03.000 INFO:teuthology.orchestra.run.smithi166.stderr: 2022-03-07T23:30:03.005 INFO:teuthology.orchestra.run.smithi078.stderr: 2022-03-07T23:30:03.005 INFO:teuthology.orchestra.run.smithi078.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-07T23:30:03.005 INFO:teuthology.orchestra.run.smithi078.stderr: 2022-03-07T23:30:03.006 INFO:teuthology.orchestra.run.smithi170.stderr: 2022-03-07T23:30:03.006 INFO:teuthology.orchestra.run.smithi170.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-07T23:30:03.006 INFO:teuthology.orchestra.run.smithi170.stderr: 2022-03-07T23:30:03.247 INFO:teuthology.orchestra.run.smithi170.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-07T23:30:03.331 INFO:teuthology.orchestra.run.smithi166.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-07T23:30:03.401 INFO:teuthology.orchestra.run.smithi166.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-07T23:30:03.483 INFO:teuthology.orchestra.run.smithi166.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-07T23:30:03.657 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-07T23:30:03.729 INFO:teuthology.orchestra.run.smithi166.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-07T23:30:03.732 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-07T23:30:03.736 INFO:teuthology.orchestra.run.smithi170.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-07T23:30:03.815 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-07T23:30:03.818 INFO:teuthology.orchestra.run.smithi170.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-07T23:30:03.897 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-07T23:30:03.900 INFO:teuthology.orchestra.run.smithi170.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-07T23:30:04.790 INFO:teuthology.orchestra.run.smithi166.stdout:Reading package lists... 2022-03-07T23:30:04.957 INFO:teuthology.orchestra.run.smithi166.stdout:Building dependency tree... 2022-03-07T23:30:04.958 INFO:teuthology.orchestra.run.smithi166.stdout:Reading state information... 2022-03-07T23:30:05.045 INFO:teuthology.orchestra.run.smithi166.stdout:81 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-07T23:30:05.061 INFO:teuthology.orchestra.run.smithi166.stderr: 2022-03-07T23:30:05.062 INFO:teuthology.orchestra.run.smithi166.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-07T23:30:05.062 INFO:teuthology.orchestra.run.smithi166.stderr: 2022-03-07T23:30:05.083 INFO:teuthology.orchestra.run.smithi170.stdout:Reading package lists... 2022-03-07T23:30:05.114 INFO:teuthology.orchestra.run.smithi166.stdout:Reading package lists... 2022-03-07T23:30:05.128 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2022-03-07T23:30:05.256 INFO:teuthology.orchestra.run.smithi170.stdout:Building dependency tree... 2022-03-07T23:30:05.257 INFO:teuthology.orchestra.run.smithi170.stdout:Reading state information... 2022-03-07T23:30:05.289 INFO:teuthology.orchestra.run.smithi166.stdout:Building dependency tree... 2022-03-07T23:30:05.289 INFO:teuthology.orchestra.run.smithi166.stdout:Reading state information... 2022-03-07T23:30:05.299 INFO:teuthology.orchestra.run.smithi078.stdout:Building dependency tree... 2022-03-07T23:30:05.300 INFO:teuthology.orchestra.run.smithi078.stdout:Reading state information... 2022-03-07T23:30:05.350 INFO:teuthology.orchestra.run.smithi170.stdout:81 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-07T23:30:05.365 INFO:teuthology.orchestra.run.smithi170.stderr: 2022-03-07T23:30:05.366 INFO:teuthology.orchestra.run.smithi170.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-07T23:30:05.366 INFO:teuthology.orchestra.run.smithi170.stderr: 2022-03-07T23:30:05.390 INFO:teuthology.orchestra.run.smithi078.stdout:81 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-07T23:30:05.407 INFO:teuthology.orchestra.run.smithi078.stderr: 2022-03-07T23:30:05.408 INFO:teuthology.orchestra.run.smithi078.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-07T23:30:05.408 INFO:teuthology.orchestra.run.smithi078.stderr: 2022-03-07T23:30:05.419 INFO:teuthology.orchestra.run.smithi170.stdout:Reading package lists... 2022-03-07T23:30:05.461 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2022-03-07T23:30:05.473 INFO:teuthology.orchestra.run.smithi166.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-07T23:30:05.473 INFO:teuthology.orchestra.run.smithi166.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-07T23:30:05.474 INFO:teuthology.orchestra.run.smithi166.stdout:The following NEW packages will be installed: 2022-03-07T23:30:05.474 INFO:teuthology.orchestra.run.smithi166.stdout: apt-transport-https 2022-03-07T23:30:05.577 INFO:teuthology.orchestra.run.smithi170.stdout:Building dependency tree... 2022-03-07T23:30:05.578 INFO:teuthology.orchestra.run.smithi170.stdout:Reading state information... 2022-03-07T23:30:05.635 INFO:teuthology.orchestra.run.smithi078.stdout:Building dependency tree... 2022-03-07T23:30:05.636 INFO:teuthology.orchestra.run.smithi078.stdout:Reading state information... 2022-03-07T23:30:05.657 INFO:teuthology.orchestra.run.smithi166.stdout:0 upgraded, 1 newly installed, 0 to remove and 81 not upgraded. 2022-03-07T23:30:05.658 INFO:teuthology.orchestra.run.smithi166.stdout:Need to get 4,680 B of archives. 2022-03-07T23:30:05.658 INFO:teuthology.orchestra.run.smithi166.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-07T23:30:05.658 INFO:teuthology.orchestra.run.smithi166.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-07T23:30:05.738 INFO:teuthology.orchestra.run.smithi170.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-07T23:30:05.738 INFO:teuthology.orchestra.run.smithi170.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-07T23:30:05.738 INFO:teuthology.orchestra.run.smithi170.stdout:The following NEW packages will be installed: 2022-03-07T23:30:05.739 INFO:teuthology.orchestra.run.smithi170.stdout: apt-transport-https 2022-03-07T23:30:05.815 INFO:teuthology.orchestra.run.smithi078.stdout:ca-certificates is already the newest version (20210119~20.04.2). 2022-03-07T23:30:05.816 INFO:teuthology.orchestra.run.smithi078.stdout:curl is already the newest version (7.68.0-1ubuntu2.7). 2022-03-07T23:30:05.816 INFO:teuthology.orchestra.run.smithi078.stdout:The following NEW packages will be installed: 2022-03-07T23:30:05.817 INFO:teuthology.orchestra.run.smithi078.stdout: apt-transport-https 2022-03-07T23:30:05.891 INFO:teuthology.orchestra.run.smithi166.stderr:debconf: unable to initialize frontend: Dialog 2022-03-07T23:30:05.891 INFO:teuthology.orchestra.run.smithi166.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-07T23:30:05.891 INFO:teuthology.orchestra.run.smithi166.stderr:debconf: falling back to frontend: Readline 2022-03-07T23:30:05.916 INFO:teuthology.orchestra.run.smithi166.stderr:debconf: unable to initialize frontend: Readline 2022-03-07T23:30:05.916 INFO:teuthology.orchestra.run.smithi166.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-07T23:30:05.917 INFO:teuthology.orchestra.run.smithi166.stderr:debconf: falling back to frontend: Teletype 2022-03-07T23:30:05.922 INFO:teuthology.orchestra.run.smithi166.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-07T23:30:05.925 INFO:teuthology.orchestra.run.smithi170.stdout:0 upgraded, 1 newly installed, 0 to remove and 81 not upgraded. 2022-03-07T23:30:05.925 INFO:teuthology.orchestra.run.smithi170.stdout:Need to get 4,680 B of archives. 2022-03-07T23:30:05.926 INFO:teuthology.orchestra.run.smithi170.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-07T23:30:05.926 INFO:teuthology.orchestra.run.smithi170.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-07T23:30:05.972 INFO:teuthology.orchestra.run.smithi166.stdout:Fetched 4,680 B in 0s (26.7 kB/s) 2022-03-07T23:30:05.999 INFO:teuthology.orchestra.run.smithi078.stdout:0 upgraded, 1 newly installed, 0 to remove and 81 not upgraded. 2022-03-07T23:30:06.000 INFO:teuthology.orchestra.run.smithi078.stdout:Need to get 4,680 B of archives. 2022-03-07T23:30:06.000 INFO:teuthology.orchestra.run.smithi078.stdout:After this operation, 162 kB of additional disk space will be used. 2022-03-07T23:30:06.001 INFO:teuthology.orchestra.run.smithi078.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.6 [4,680 B] 2022-03-07T23:30:06.078 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package apt-transport-https. 2022-03-07T23:30:06.119 INFO:teuthology.orchestra.run.smithi166.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-07T23:30:06.122 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-07T23:30:06.159 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-07T23:30:06.161 INFO:teuthology.orchestra.run.smithi170.stderr:debconf: unable to initialize frontend: Dialog 2022-03-07T23:30:06.161 INFO:teuthology.orchestra.run.smithi170.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-07T23:30:06.161 INFO:teuthology.orchestra.run.smithi170.stderr:debconf: falling back to frontend: Readline 2022-03-07T23:30:06.178 INFO:teuthology.orchestra.run.smithi170.stderr:debconf: unable to initialize frontend: Readline 2022-03-07T23:30:06.179 INFO:teuthology.orchestra.run.smithi170.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-07T23:30:06.179 INFO:teuthology.orchestra.run.smithi170.stderr:debconf: falling back to frontend: Teletype 2022-03-07T23:30:06.184 INFO:teuthology.orchestra.run.smithi170.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-07T23:30:06.232 INFO:teuthology.orchestra.run.smithi170.stdout:Fetched 4,680 B in 0s (26.3 kB/s) 2022-03-07T23:30:06.235 INFO:teuthology.orchestra.run.smithi078.stderr:debconf: unable to initialize frontend: Dialog 2022-03-07T23:30:06.235 INFO:teuthology.orchestra.run.smithi078.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-07T23:30:06.236 INFO:teuthology.orchestra.run.smithi078.stderr:debconf: falling back to frontend: Readline 2022-03-07T23:30:06.256 INFO:teuthology.orchestra.run.smithi078.stderr:debconf: unable to initialize frontend: Readline 2022-03-07T23:30:06.256 INFO:teuthology.orchestra.run.smithi078.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-07T23:30:06.256 INFO:teuthology.orchestra.run.smithi078.stderr:debconf: falling back to frontend: Teletype 2022-03-07T23:30:06.262 INFO:teuthology.orchestra.run.smithi078.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-07T23:30:06.315 INFO:teuthology.orchestra.run.smithi078.stdout:Fetched 4,680 B in 0s (26.8 kB/s) 2022-03-07T23:30:06.347 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package apt-transport-https. 2022-03-07T23:30:06.393 INFO:teuthology.orchestra.run.smithi170.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-07T23:30:06.396 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-07T23:30:06.413 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package apt-transport-https. 2022-03-07T23:30:06.450 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-07T23:30:06.458 INFO:teuthology.orchestra.run.smithi078.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109240 files and directories currently installed.) 2022-03-07T23:30:06.461 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../apt-transport-https_2.0.6_all.deb ... 2022-03-07T23:30:06.462 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-07T23:30:06.503 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking apt-transport-https (2.0.6) ... 2022-03-07T23:30:06.746 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-07T23:30:06.758 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up apt-transport-https (2.0.6) ... 2022-03-07T23:30:07.051 INFO:teuthology.orchestra.run.smithi166.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-07T23:30:07.066 INFO:teuthology.orchestra.run.smithi166.stderr: 2022-03-07T23:30:07.066 INFO:teuthology.orchestra.run.smithi166.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-07T23:30:07.066 INFO:teuthology.orchestra.run.smithi166.stderr: 2022-03-07T23:30:07.193 INFO:teuthology.orchestra.run.smithi166.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-07T23:30:07.244 INFO:teuthology.orchestra.run.smithi170.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-07T23:30:07.259 INFO:teuthology.orchestra.run.smithi170.stderr: 2022-03-07T23:30:07.259 INFO:teuthology.orchestra.run.smithi170.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-07T23:30:07.259 INFO:teuthology.orchestra.run.smithi170.stderr: 2022-03-07T23:30:07.265 INFO:teuthology.orchestra.run.smithi078.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-03-07T23:30:07.278 INFO:teuthology.orchestra.run.smithi078.stderr: 2022-03-07T23:30:07.278 INFO:teuthology.orchestra.run.smithi078.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-07T23:30:07.279 INFO:teuthology.orchestra.run.smithi078.stderr: 2022-03-07T23:30:07.306 INFO:teuthology.orchestra.run.smithi166.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-07T23:30:07.388 INFO:teuthology.orchestra.run.smithi166.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-07T23:30:07.470 INFO:teuthology.orchestra.run.smithi166.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-07T23:30:07.480 INFO:teuthology.orchestra.run.smithi166.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-07T23:30:07.512 INFO:teuthology.orchestra.run.smithi170.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-07T23:30:07.512 INFO:teuthology.orchestra.run.smithi170.stdout:Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-07T23:30:07.544 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-03-07T23:30:07.544 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-03-07T23:30:07.582 INFO:teuthology.orchestra.run.smithi170.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-07T23:30:07.607 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-03-07T23:30:07.610 INFO:teuthology.orchestra.run.smithi170.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-07T23:30:07.664 INFO:teuthology.orchestra.run.smithi170.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-07T23:30:07.690 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-03-07T23:30:07.803 INFO:teuthology.orchestra.run.smithi078.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-03-07T23:30:07.844 INFO:teuthology.orchestra.run.smithi166.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-07T23:30:07.981 INFO:teuthology.orchestra.run.smithi166.stdout:Fetched 63.3 kB in 1s (82.2 kB/s) 2022-03-07T23:30:08.008 INFO:teuthology.orchestra.run.smithi170.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-07T23:30:08.167 INFO:teuthology.orchestra.run.smithi078.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [53.9 kB] 2022-03-07T23:30:08.198 INFO:teuthology.orchestra.run.smithi170.stdout:Fetched 63.3 kB in 1s (79.7 kB/s) 2022-03-07T23:30:08.302 INFO:teuthology.orchestra.run.smithi078.stdout:Fetched 63.3 kB in 1s (72.5 kB/s) 2022-03-07T23:30:08.832 INFO:teuthology.orchestra.run.smithi166.stdout:Reading package lists... 2022-03-07T23:30:09.006 INFO:teuthology.orchestra.run.smithi166.stdout:Building dependency tree... 2022-03-07T23:30:09.007 INFO:teuthology.orchestra.run.smithi166.stdout:Reading state information... 2022-03-07T23:30:09.061 INFO:teuthology.orchestra.run.smithi170.stdout:Reading package lists... 2022-03-07T23:30:09.099 INFO:teuthology.orchestra.run.smithi166.stdout:81 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-07T23:30:09.116 INFO:teuthology.orchestra.run.smithi166.stderr: 2022-03-07T23:30:09.116 INFO:teuthology.orchestra.run.smithi166.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-07T23:30:09.117 INFO:teuthology.orchestra.run.smithi166.stderr: 2022-03-07T23:30:09.146 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2022-03-07T23:30:09.168 INFO:teuthology.orchestra.run.smithi166.stdout:Reading package lists... 2022-03-07T23:30:09.229 INFO:teuthology.orchestra.run.smithi170.stdout:Building dependency tree... 2022-03-07T23:30:09.230 INFO:teuthology.orchestra.run.smithi170.stdout:Reading state information... 2022-03-07T23:30:09.320 INFO:teuthology.orchestra.run.smithi078.stdout:Building dependency tree... 2022-03-07T23:30:09.321 INFO:teuthology.orchestra.run.smithi078.stdout:Reading state information... 2022-03-07T23:30:09.327 INFO:teuthology.orchestra.run.smithi170.stdout:81 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-07T23:30:09.330 INFO:teuthology.orchestra.run.smithi166.stdout:Building dependency tree... 2022-03-07T23:30:09.331 INFO:teuthology.orchestra.run.smithi166.stdout:Reading state information... 2022-03-07T23:30:09.338 INFO:teuthology.orchestra.run.smithi170.stderr: 2022-03-07T23:30:09.338 INFO:teuthology.orchestra.run.smithi170.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-07T23:30:09.338 INFO:teuthology.orchestra.run.smithi170.stderr: 2022-03-07T23:30:09.393 INFO:teuthology.orchestra.run.smithi170.stdout:Reading package lists... 2022-03-07T23:30:09.413 INFO:teuthology.orchestra.run.smithi078.stdout:81 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-03-07T23:30:09.431 INFO:teuthology.orchestra.run.smithi078.stderr: 2022-03-07T23:30:09.431 INFO:teuthology.orchestra.run.smithi078.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-03-07T23:30:09.431 INFO:teuthology.orchestra.run.smithi078.stderr: 2022-03-07T23:30:09.463 INFO:teuthology.orchestra.run.smithi166.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-07T23:30:09.463 INFO:teuthology.orchestra.run.smithi166.stdout:bridge-utils set to manually installed. 2022-03-07T23:30:09.463 INFO:teuthology.orchestra.run.smithi166.stdout:The following additional packages will be installed: 2022-03-07T23:30:09.464 INFO:teuthology.orchestra.run.smithi166.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-07T23:30:09.464 INFO:teuthology.orchestra.run.smithi166.stdout:Suggested packages: 2022-03-07T23:30:09.464 INFO:teuthology.orchestra.run.smithi166.stdout: nftables 2022-03-07T23:30:09.486 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2022-03-07T23:30:09.514 INFO:teuthology.orchestra.run.smithi166.stdout:The following NEW packages will be installed: 2022-03-07T23:30:09.515 INFO:teuthology.orchestra.run.smithi166.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-07T23:30:09.562 INFO:teuthology.orchestra.run.smithi170.stdout:Building dependency tree... 2022-03-07T23:30:09.562 INFO:teuthology.orchestra.run.smithi170.stdout:Reading state information... 2022-03-07T23:30:09.656 INFO:teuthology.orchestra.run.smithi078.stdout:Building dependency tree... 2022-03-07T23:30:09.657 INFO:teuthology.orchestra.run.smithi078.stdout:Reading state information... 2022-03-07T23:30:09.692 INFO:teuthology.orchestra.run.smithi170.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-07T23:30:09.692 INFO:teuthology.orchestra.run.smithi170.stdout:bridge-utils set to manually installed. 2022-03-07T23:30:09.693 INFO:teuthology.orchestra.run.smithi170.stdout:The following additional packages will be installed: 2022-03-07T23:30:09.693 INFO:teuthology.orchestra.run.smithi170.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-07T23:30:09.694 INFO:teuthology.orchestra.run.smithi170.stdout:Suggested packages: 2022-03-07T23:30:09.694 INFO:teuthology.orchestra.run.smithi170.stdout: nftables 2022-03-07T23:30:09.705 INFO:teuthology.orchestra.run.smithi166.stdout:0 upgraded, 8 newly installed, 0 to remove and 81 not upgraded. 2022-03-07T23:30:09.705 INFO:teuthology.orchestra.run.smithi166.stdout:Need to get 73.6 MB of archives. 2022-03-07T23:30:09.706 INFO:teuthology.orchestra.run.smithi166.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-07T23:30:09.706 INFO:teuthology.orchestra.run.smithi166.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-07T23:30:09.738 INFO:teuthology.orchestra.run.smithi166.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-07T23:30:09.742 INFO:teuthology.orchestra.run.smithi170.stdout:The following NEW packages will be installed: 2022-03-07T23:30:09.743 INFO:teuthology.orchestra.run.smithi170.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-07T23:30:09.785 INFO:teuthology.orchestra.run.smithi078.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-03-07T23:30:09.786 INFO:teuthology.orchestra.run.smithi078.stdout:bridge-utils set to manually installed. 2022-03-07T23:30:09.786 INFO:teuthology.orchestra.run.smithi078.stdout:The following additional packages will be installed: 2022-03-07T23:30:09.787 INFO:teuthology.orchestra.run.smithi078.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-03-07T23:30:09.787 INFO:teuthology.orchestra.run.smithi078.stdout:Suggested packages: 2022-03-07T23:30:09.788 INFO:teuthology.orchestra.run.smithi078.stdout: nftables 2022-03-07T23:30:09.832 INFO:teuthology.orchestra.run.smithi078.stdout:The following NEW packages will be installed: 2022-03-07T23:30:09.834 INFO:teuthology.orchestra.run.smithi078.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-03-07T23:30:09.881 INFO:teuthology.orchestra.run.smithi166.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-07T23:30:09.945 INFO:teuthology.orchestra.run.smithi170.stdout:0 upgraded, 8 newly installed, 0 to remove and 81 not upgraded. 2022-03-07T23:30:09.946 INFO:teuthology.orchestra.run.smithi170.stdout:Need to get 73.6 MB of archives. 2022-03-07T23:30:09.946 INFO:teuthology.orchestra.run.smithi170.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-07T23:30:09.946 INFO:teuthology.orchestra.run.smithi170.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-07T23:30:09.996 INFO:teuthology.orchestra.run.smithi166.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-07T23:30:10.001 INFO:teuthology.orchestra.run.smithi170.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-07T23:30:10.014 INFO:teuthology.orchestra.run.smithi166.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-07T23:30:10.029 INFO:teuthology.orchestra.run.smithi078.stdout:0 upgraded, 8 newly installed, 0 to remove and 81 not upgraded. 2022-03-07T23:30:10.029 INFO:teuthology.orchestra.run.smithi078.stdout:Need to get 73.6 MB of archives. 2022-03-07T23:30:10.029 INFO:teuthology.orchestra.run.smithi078.stdout:After this operation, 321 MB of additional disk space will be used. 2022-03-07T23:30:10.030 INFO:teuthology.orchestra.run.smithi078.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-03-07T23:30:10.078 INFO:teuthology.orchestra.run.smithi078.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.19.0-00 [11.2 MB] 2022-03-07T23:30:10.136 INFO:teuthology.orchestra.run.smithi170.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-07T23:30:10.236 INFO:teuthology.orchestra.run.smithi078.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-03-07T23:30:10.276 INFO:teuthology.orchestra.run.smithi170.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-07T23:30:10.341 INFO:teuthology.orchestra.run.smithi170.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-07T23:30:10.361 INFO:teuthology.orchestra.run.smithi078.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-03-07T23:30:10.660 INFO:teuthology.orchestra.run.smithi166.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-07T23:30:11.000 INFO:teuthology.orchestra.run.smithi166.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-07T23:30:11.003 INFO:teuthology.orchestra.run.smithi170.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-07T23:30:11.155 INFO:teuthology.orchestra.run.smithi166.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-07T23:30:11.380 INFO:teuthology.orchestra.run.smithi170.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-07T23:30:11.600 INFO:teuthology.orchestra.run.smithi170.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-07T23:30:11.664 INFO:teuthology.orchestra.run.smithi166.stderr:debconf: unable to initialize frontend: Dialog 2022-03-07T23:30:11.664 INFO:teuthology.orchestra.run.smithi166.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-07T23:30:11.665 INFO:teuthology.orchestra.run.smithi166.stderr:debconf: falling back to frontend: Readline 2022-03-07T23:30:11.672 INFO:teuthology.orchestra.run.smithi166.stderr:debconf: unable to initialize frontend: Readline 2022-03-07T23:30:11.672 INFO:teuthology.orchestra.run.smithi166.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-07T23:30:11.672 INFO:teuthology.orchestra.run.smithi166.stderr:debconf: falling back to frontend: Teletype 2022-03-07T23:30:11.677 INFO:teuthology.orchestra.run.smithi166.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-07T23:30:11.729 INFO:teuthology.orchestra.run.smithi166.stdout:Fetched 73.6 MB in 2s (38.3 MB/s) 2022-03-07T23:30:11.828 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package conntrack. 2022-03-07T23:30:12.551 INFO:teuthology.orchestra.run.smithi078.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-03-07T23:30:12.553 INFO:teuthology.orchestra.run.smithi170.stderr:debconf: unable to initialize frontend: Dialog 2022-03-07T23:30:12.553 INFO:teuthology.orchestra.run.smithi170.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-07T23:30:12.554 INFO:teuthology.orchestra.run.smithi170.stderr:debconf: falling back to frontend: Readline 2022-03-07T23:30:12.554 INFO:teuthology.orchestra.run.smithi170.stderr:debconf: unable to initialize frontend: Readline 2022-03-07T23:30:12.554 INFO:teuthology.orchestra.run.smithi170.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-07T23:30:12.554 INFO:teuthology.orchestra.run.smithi170.stderr:debconf: falling back to frontend: Teletype 2022-03-07T23:30:12.554 INFO:teuthology.orchestra.run.smithi170.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-07T23:30:12.557 INFO:teuthology.orchestra.run.smithi166.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-07T23:30:12.557 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-07T23:30:12.558 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-07T23:30:12.558 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package cri-tools. 2022-03-07T23:30:12.558 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-07T23:30:12.558 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-07T23:30:12.559 INFO:teuthology.orchestra.run.smithi170.stdout:Fetched 73.6 MB in 2s (35.8 MB/s) 2022-03-07T23:30:12.559 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package conntrack. 2022-03-07T23:30:12.559 INFO:teuthology.orchestra.run.smithi170.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-07T23:30:12.560 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-07T23:30:12.560 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-07T23:30:12.600 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package cri-tools. 2022-03-07T23:30:12.613 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-07T23:30:12.650 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-07T23:30:12.712 INFO:teuthology.orchestra.run.smithi078.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.23.4-00 [19.5 MB] 2022-03-07T23:30:13.042 INFO:teuthology.orchestra.run.smithi078.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.23.4-00 [8,927 kB] 2022-03-07T23:30:13.140 INFO:teuthology.orchestra.run.smithi078.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.23.4-00 [8,583 kB] 2022-03-07T23:30:13.420 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package ebtables. 2022-03-07T23:30:13.434 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-07T23:30:13.495 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-07T23:30:13.513 INFO:teuthology.orchestra.run.smithi078.stderr:debconf: unable to initialize frontend: Dialog 2022-03-07T23:30:13.514 INFO:teuthology.orchestra.run.smithi078.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-03-07T23:30:13.514 INFO:teuthology.orchestra.run.smithi078.stderr:debconf: falling back to frontend: Readline 2022-03-07T23:30:13.522 INFO:teuthology.orchestra.run.smithi078.stderr:debconf: unable to initialize frontend: Readline 2022-03-07T23:30:13.523 INFO:teuthology.orchestra.run.smithi078.stderr:debconf: (This frontend requires a controlling tty.) 2022-03-07T23:30:13.523 INFO:teuthology.orchestra.run.smithi078.stderr:debconf: falling back to frontend: Teletype 2022-03-07T23:30:13.529 INFO:teuthology.orchestra.run.smithi078.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-03-07T23:30:13.580 INFO:teuthology.orchestra.run.smithi078.stdout:Fetched 73.6 MB in 3s (21.3 MB/s) 2022-03-07T23:30:13.678 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package conntrack. 2022-03-07T23:30:13.722 INFO:teuthology.orchestra.run.smithi078.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109244 files and directories currently installed.) 2022-03-07T23:30:13.725 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-03-07T23:30:13.728 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package ebtables. 2022-03-07T23:30:13.741 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-07T23:30:13.768 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-03-07T23:30:13.785 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-07T23:30:13.789 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-07T23:30:13.803 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-07T23:30:13.840 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-07T23:30:14.071 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package cri-tools. 2022-03-07T23:30:14.083 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../1-cri-tools_1.19.0-00_amd64.deb ... 2022-03-07T23:30:14.097 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-07T23:30:14.111 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-07T23:30:14.120 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking cri-tools (1.19.0-00) ... 2022-03-07T23:30:14.147 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-07T23:30:15.182 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ebtables. 2022-03-07T23:30:15.194 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-03-07T23:30:15.248 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-03-07T23:30:15.325 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package socat. 2022-03-07T23:30:15.339 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-07T23:30:15.384 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-07T23:30:15.526 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package kubernetes-cni. 2022-03-07T23:30:15.539 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-03-07T23:30:15.576 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-03-07T23:30:15.616 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package socat. 2022-03-07T23:30:15.631 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-07T23:30:15.675 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-07T23:30:15.795 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package kubelet. 2022-03-07T23:30:15.809 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-07T23:30:15.845 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-07T23:30:16.054 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package kubelet. 2022-03-07T23:30:16.067 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-07T23:30:16.128 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-07T23:30:17.095 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package socat. 2022-03-07T23:30:17.109 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-03-07T23:30:17.153 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking socat (1.7.3.3-2) ... 2022-03-07T23:30:17.523 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package kubelet. 2022-03-07T23:30:17.538 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../5-kubelet_1.23.4-00_amd64.deb ... 2022-03-07T23:30:17.581 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking kubelet (1.23.4-00) ... 2022-03-07T23:30:18.247 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package kubectl. 2022-03-07T23:30:18.260 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-07T23:30:18.314 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-07T23:30:18.656 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package kubectl. 2022-03-07T23:30:18.672 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-07T23:30:18.740 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-07T23:30:19.542 INFO:teuthology.orchestra.run.smithi166.stdout:Selecting previously unselected package kubeadm. 2022-03-07T23:30:19.550 INFO:teuthology.orchestra.run.smithi166.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-07T23:30:19.592 INFO:teuthology.orchestra.run.smithi166.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-07T23:30:19.942 INFO:teuthology.orchestra.run.smithi170.stdout:Selecting previously unselected package kubeadm. 2022-03-07T23:30:19.956 INFO:teuthology.orchestra.run.smithi170.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-07T23:30:19.984 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package kubectl. 2022-03-07T23:30:19.992 INFO:teuthology.orchestra.run.smithi170.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-07T23:30:19.996 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../6-kubectl_1.23.4-00_amd64.deb ... 2022-03-07T23:30:20.034 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking kubectl (1.23.4-00) ... 2022-03-07T23:30:20.781 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-07T23:30:20.906 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-07T23:30:21.041 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-07T23:30:21.173 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-07T23:30:21.234 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package kubeadm. 2022-03-07T23:30:21.249 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../7-kubeadm_1.23.4-00_amd64.deb ... 2022-03-07T23:30:21.293 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking kubeadm (1.23.4-00) ... 2022-03-07T23:30:21.307 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-07T23:30:21.474 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-07T23:30:21.906 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-07T23:30:22.032 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-07T23:30:22.201 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-07T23:30:22.299 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-07T23:30:22.327 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-07T23:30:22.422 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-03-07T23:30:22.450 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-07T23:30:22.540 INFO:teuthology.orchestra.run.smithi166.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-07T23:30:22.559 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-07T23:30:22.563 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up kubectl (1.23.4-00) ... 2022-03-07T23:30:22.689 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-03-07T23:30:22.693 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-07T23:30:22.911 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-07T23:30:23.077 INFO:teuthology.orchestra.run.smithi166.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-07T23:30:23.395 INFO:teuthology.orchestra.run.smithi170.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-07T23:30:23.471 INFO:teuthology.orchestra.run.smithi166.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-07T23:30:23.495 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up socat (1.7.3.3-2) ... 2022-03-07T23:30:23.612 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up cri-tools (1.19.0-00) ... 2022-03-07T23:30:23.714 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-03-07T23:30:23.842 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up kubelet (1.23.4-00) ... 2022-03-07T23:30:23.847 INFO:teuthology.orchestra.run.smithi170.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-07T23:30:24.066 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-03-07T23:30:24.562 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up kubeadm (1.23.4-00) ... 2022-03-07T23:30:24.947 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-03-07T23:30:25.767 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-07T23:30:25.770 DEBUG:teuthology.orchestra.run.smithi166:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-07T23:30:25.772 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-03-07T23:30:38.679 INFO:teuthology.orchestra.run.smithi170.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-07T23:30:39.059 INFO:teuthology.orchestra.run.smithi166.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-07T23:30:39.402 INFO:teuthology.orchestra.run.smithi078.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.4 2022-03-07T23:30:42.623 INFO:teuthology.orchestra.run.smithi170.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-07T23:30:43.012 INFO:teuthology.orchestra.run.smithi166.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-07T23:30:43.378 INFO:teuthology.orchestra.run.smithi078.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.4 2022-03-07T23:30:44.912 INFO:teuthology.orchestra.run.smithi170.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-07T23:30:45.467 INFO:teuthology.orchestra.run.smithi166.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-07T23:30:45.926 INFO:teuthology.orchestra.run.smithi078.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.4 2022-03-07T23:30:48.587 INFO:teuthology.orchestra.run.smithi170.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-07T23:30:49.183 INFO:teuthology.orchestra.run.smithi166.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-07T23:30:49.359 INFO:teuthology.orchestra.run.smithi078.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.4 2022-03-07T23:30:50.011 INFO:teuthology.orchestra.run.smithi170.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-07T23:30:50.700 INFO:teuthology.orchestra.run.smithi166.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-07T23:30:50.725 INFO:teuthology.orchestra.run.smithi078.stdout:[config/images] Pulled k8s.gcr.io/pause:3.6 2022-03-07T23:30:55.785 INFO:teuthology.orchestra.run.smithi170.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-07T23:30:57.181 INFO:teuthology.orchestra.run.smithi078.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-07T23:30:57.407 INFO:teuthology.orchestra.run.smithi166.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.1-0 2022-03-07T23:30:57.834 INFO:teuthology.orchestra.run.smithi170.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-07T23:30:59.158 INFO:teuthology.orchestra.run.smithi078.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-07T23:30:59.573 INFO:teuthology.orchestra.run.smithi166.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-03-07T23:30:59.578 INFO:tasks.kubeadm:Token: abcdef.0ifgguk9k7y1ogys 2022-03-07T23:30:59.579 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi078.front.sepia.ceph.com'): '172.21.15.78', Remote(name='ubuntu@smithi166.front.sepia.ceph.com'): '172.21.15.166', Remote(name='ubuntu@smithi170.front.sepia.ceph.com'): '172.21.15.170'} 2022-03-07T23:30:59.579 DEBUG:teuthology.orchestra.run.smithi078:> sudo kubeadm init --node-name smithi078 --token abcdef.0ifgguk9k7y1ogys --pod-network-cidr 10.250.104.0/21 2022-03-07T23:30:59.777 INFO:teuthology.orchestra.run.smithi078.stdout:[init] Using Kubernetes version: v1.23.4 2022-03-07T23:30:59.777 INFO:teuthology.orchestra.run.smithi078.stdout:[preflight] Running pre-flight checks 2022-03-07T23:31:00.177 INFO:teuthology.orchestra.run.smithi078.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-03-07T23:31:00.178 INFO:teuthology.orchestra.run.smithi078.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-03-07T23:31:00.178 INFO:teuthology.orchestra.run.smithi078.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-03-07T23:31:00.495 INFO:teuthology.orchestra.run.smithi078.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-03-07T23:31:00.751 INFO:teuthology.orchestra.run.smithi078.stdout:[certs] Generating "ca" certificate and key 2022-03-07T23:31:00.871 INFO:teuthology.orchestra.run.smithi078.stdout:[certs] Generating "apiserver" certificate and key 2022-03-07T23:31:00.872 INFO:teuthology.orchestra.run.smithi078.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi078] and IPs [10.96.0.1 172.21.15.78] 2022-03-07T23:31:00.978 INFO:teuthology.orchestra.run.smithi078.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-03-07T23:31:01.482 INFO:teuthology.orchestra.run.smithi078.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-03-07T23:31:01.599 INFO:teuthology.orchestra.run.smithi078.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-03-07T23:31:01.821 INFO:teuthology.orchestra.run.smithi078.stdout:[certs] Generating "etcd/ca" certificate and key 2022-03-07T23:31:01.949 INFO:teuthology.orchestra.run.smithi078.stdout:[certs] Generating "etcd/server" certificate and key 2022-03-07T23:31:01.950 INFO:teuthology.orchestra.run.smithi078.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi078] and IPs [172.21.15.78 127.0.0.1 ::1] 2022-03-07T23:31:02.036 INFO:teuthology.orchestra.run.smithi078.stdout:[certs] Generating "etcd/peer" certificate and key 2022-03-07T23:31:02.036 INFO:teuthology.orchestra.run.smithi078.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi078] and IPs [172.21.15.78 127.0.0.1 ::1] 2022-03-07T23:31:02.146 INFO:teuthology.orchestra.run.smithi078.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-03-07T23:31:02.315 INFO:teuthology.orchestra.run.smithi078.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-03-07T23:31:02.442 INFO:teuthology.orchestra.run.smithi078.stdout:[certs] Generating "sa" key and public key 2022-03-07T23:31:02.443 INFO:teuthology.orchestra.run.smithi078.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-03-07T23:31:02.579 INFO:teuthology.orchestra.run.smithi078.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-03-07T23:31:02.722 INFO:teuthology.orchestra.run.smithi078.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-03-07T23:31:02.881 INFO:teuthology.orchestra.run.smithi078.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-03-07T23:31:03.119 INFO:teuthology.orchestra.run.smithi078.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-03-07T23:31:03.134 INFO:teuthology.orchestra.run.smithi078.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-07T23:31:03.136 INFO:teuthology.orchestra.run.smithi078.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-07T23:31:03.136 INFO:teuthology.orchestra.run.smithi078.stdout:[kubelet-start] Starting the kubelet 2022-03-07T23:31:03.359 INFO:teuthology.orchestra.run.smithi078.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-03-07T23:31:03.359 INFO:teuthology.orchestra.run.smithi078.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-03-07T23:31:03.361 INFO:teuthology.orchestra.run.smithi078.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-03-07T23:31:03.362 INFO:teuthology.orchestra.run.smithi078.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-03-07T23:31:03.362 INFO:teuthology.orchestra.run.smithi078.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-03-07T23:31:03.364 INFO:teuthology.orchestra.run.smithi078.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-03-07T23:31:19.884 INFO:teuthology.orchestra.run.smithi078.stdout:[apiclient] All control plane components are healthy after 16.519470 seconds 2022-03-07T23:31:19.884 INFO:teuthology.orchestra.run.smithi078.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-03-07T23:31:19.902 INFO:teuthology.orchestra.run.smithi078.stdout:[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster 2022-03-07T23:31:19.903 INFO:teuthology.orchestra.run.smithi078.stdout:NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently. 2022-03-07T23:31:20.461 INFO:teuthology.orchestra.run.smithi078.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-03-07T23:31:20.461 INFO:teuthology.orchestra.run.smithi078.stdout:[mark-control-plane] Marking the node smithi078 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-03-07T23:31:20.462 INFO:teuthology.orchestra.run.smithi078.stdout:[mark-control-plane] Marking the node smithi078 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] 2022-03-07T23:31:21.049 INFO:teuthology.orchestra.run.smithi078.stdout:[bootstrap-token] Using token: abcdef.0ifgguk9k7y1ogys 2022-03-07T23:31:21.050 INFO:teuthology.orchestra.run.smithi078.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-03-07T23:31:21.055 INFO:teuthology.orchestra.run.smithi078.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-03-07T23:31:21.069 INFO:teuthology.orchestra.run.smithi078.stdout:[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-03-07T23:31:21.078 INFO:teuthology.orchestra.run.smithi078.stdout:[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-03-07T23:31:21.086 INFO:teuthology.orchestra.run.smithi078.stdout:[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-03-07T23:31:21.094 INFO:teuthology.orchestra.run.smithi078.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-03-07T23:31:21.136 INFO:teuthology.orchestra.run.smithi078.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-03-07T23:31:21.588 INFO:teuthology.orchestra.run.smithi078.stdout:[addons] Applied essential addon: CoreDNS 2022-03-07T23:31:21.745 INFO:teuthology.orchestra.run.smithi078.stdout:[addons] Applied essential addon: kube-proxy 2022-03-07T23:31:21.746 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:31:21.746 INFO:teuthology.orchestra.run.smithi078.stdout:Your Kubernetes control-plane has initialized successfully! 2022-03-07T23:31:21.746 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:31:21.747 INFO:teuthology.orchestra.run.smithi078.stdout:To start using your cluster, you need to run the following as a regular user: 2022-03-07T23:31:21.747 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:31:21.747 INFO:teuthology.orchestra.run.smithi078.stdout: mkdir -p $HOME/.kube 2022-03-07T23:31:21.747 INFO:teuthology.orchestra.run.smithi078.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-03-07T23:31:21.748 INFO:teuthology.orchestra.run.smithi078.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-03-07T23:31:21.748 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:31:21.748 INFO:teuthology.orchestra.run.smithi078.stdout:Alternatively, if you are the root user, you can run: 2022-03-07T23:31:21.748 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:31:21.749 INFO:teuthology.orchestra.run.smithi078.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-03-07T23:31:21.749 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:31:21.749 INFO:teuthology.orchestra.run.smithi078.stdout:You should now deploy a pod network to the cluster. 2022-03-07T23:31:21.749 INFO:teuthology.orchestra.run.smithi078.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-03-07T23:31:21.750 INFO:teuthology.orchestra.run.smithi078.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-03-07T23:31:21.750 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:31:21.750 INFO:teuthology.orchestra.run.smithi078.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-03-07T23:31:21.750 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:31:21.750 INFO:teuthology.orchestra.run.smithi078.stdout:kubeadm join 172.21.15.78:6443 --token abcdef.0ifgguk9k7y1ogys \ 2022-03-07T23:31:21.751 INFO:teuthology.orchestra.run.smithi078.stdout: --discovery-token-ca-cert-hash sha256:2b2f49371956c5b5078bc1c5f8a3f583f179f58423d625793f1234e1b4848c99 2022-03-07T23:31:21.752 DEBUG:teuthology.orchestra.run.smithi166:> sudo kubeadm join 172.21.15.78:6443 --node-name smithi166 --token abcdef.0ifgguk9k7y1ogys --discovery-token-unsafe-skip-ca-verification 2022-03-07T23:31:21.756 DEBUG:teuthology.orchestra.run.smithi170:> sudo kubeadm join 172.21.15.78:6443 --node-name smithi170 --token abcdef.0ifgguk9k7y1ogys --discovery-token-unsafe-skip-ca-verification 2022-03-07T23:31:21.792 INFO:teuthology.orchestra.run.smithi170.stdout:[preflight] Running pre-flight checks 2022-03-07T23:31:21.797 INFO:teuthology.orchestra.run.smithi166.stdout:[preflight] Running pre-flight checks 2022-03-07T23:31:34.539 INFO:teuthology.orchestra.run.smithi170.stdout:[preflight] Reading configuration from the cluster... 2022-03-07T23:31:34.539 INFO:teuthology.orchestra.run.smithi170.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-07T23:31:34.731 INFO:teuthology.orchestra.run.smithi166.stdout:[preflight] Reading configuration from the cluster... 2022-03-07T23:31:34.731 INFO:teuthology.orchestra.run.smithi166.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-03-07T23:31:34.901 INFO:teuthology.orchestra.run.smithi170.stderr:W0307 23:31:34.903215 10662 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-07T23:31:34.920 INFO:teuthology.orchestra.run.smithi170.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-07T23:31:34.920 INFO:teuthology.orchestra.run.smithi170.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-07T23:31:34.920 INFO:teuthology.orchestra.run.smithi170.stdout:[kubelet-start] Starting the kubelet 2022-03-07T23:31:35.093 INFO:teuthology.orchestra.run.smithi166.stderr:W0307 23:31:35.094321 10683 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf 2022-03-07T23:31:35.111 INFO:teuthology.orchestra.run.smithi166.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-03-07T23:31:35.111 INFO:teuthology.orchestra.run.smithi166.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-03-07T23:31:35.111 INFO:teuthology.orchestra.run.smithi166.stdout:[kubelet-start] Starting the kubelet 2022-03-07T23:31:35.151 INFO:teuthology.orchestra.run.smithi170.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-07T23:31:35.343 INFO:teuthology.orchestra.run.smithi166.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-03-07T23:31:40.869 INFO:teuthology.orchestra.run.smithi166.stdout: 2022-03-07T23:31:40.870 INFO:teuthology.orchestra.run.smithi166.stdout:This node has joined the cluster: 2022-03-07T23:31:40.870 INFO:teuthology.orchestra.run.smithi166.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-07T23:31:40.870 INFO:teuthology.orchestra.run.smithi166.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-07T23:31:40.871 INFO:teuthology.orchestra.run.smithi166.stdout: 2022-03-07T23:31:40.871 INFO:teuthology.orchestra.run.smithi166.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-07T23:31:40.871 INFO:teuthology.orchestra.run.smithi166.stdout: 2022-03-07T23:31:48.681 INFO:teuthology.orchestra.run.smithi170.stdout: 2022-03-07T23:31:48.681 INFO:teuthology.orchestra.run.smithi170.stdout:This node has joined the cluster: 2022-03-07T23:31:48.681 INFO:teuthology.orchestra.run.smithi170.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-03-07T23:31:48.682 INFO:teuthology.orchestra.run.smithi170.stdout:* The Kubelet was informed of the new secure connection details. 2022-03-07T23:31:48.682 INFO:teuthology.orchestra.run.smithi170.stdout: 2022-03-07T23:31:48.682 INFO:teuthology.orchestra.run.smithi170.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-03-07T23:31:48.682 INFO:teuthology.orchestra.run.smithi170.stdout: 2022-03-07T23:31:48.683 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2022-03-07T23:31:48.683 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-03-07T23:31:48.698 INFO:tasks.kubeadm:Setting up kubectl 2022-03-07T23:31:48.698 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-07T23:31:48.711 DEBUG:teuthology.orchestra.run.smithi166:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-07T23:31:48.727 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-03-07T23:31:48.744 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2022-03-07T23:31:48.744 DEBUG:teuthology.orchestra.run.smithi078:> dd of=.kube/config 2022-03-07T23:31:48.761 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2022-03-07T23:31:48.761 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/root/.kube/config 2022-03-07T23:31:48.774 DEBUG:teuthology.orchestra.run.smithi166:> set -ex 2022-03-07T23:31:48.775 DEBUG:teuthology.orchestra.run.smithi166:> dd of=.kube/config 2022-03-07T23:31:48.782 DEBUG:teuthology.orchestra.run.smithi166:> set -ex 2022-03-07T23:31:48.783 DEBUG:teuthology.orchestra.run.smithi166:> sudo dd of=/root/.kube/config 2022-03-07T23:31:48.797 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2022-03-07T23:31:48.797 DEBUG:teuthology.orchestra.run.smithi170:> dd of=.kube/config 2022-03-07T23:31:48.805 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2022-03-07T23:31:48.805 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/root/.kube/config 2022-03-07T23:31:48.818 DEBUG:teuthology.orchestra.run.smithi078:> curl https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml 2022-03-07T23:31:48.834 INFO:teuthology.orchestra.run.smithi078.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-03-07T23:31:48.835 INFO:teuthology.orchestra.run.smithi078.stderr: Dload Upload Total Spent Left Speed 2022-03-07T23:31:48.962 INFO:teuthology.orchestra.run.smithi078.stdout:--- 2022-03-07T23:31:48.962 INFO:teuthology.orchestra.run.smithi078.stdout:apiVersion: policy/v1beta1 2022-03-07T23:31:48.962 INFO:teuthology.orchestra.run.smithi078.stdout:kind: PodSecurityPolicy 2022-03-07T23:31:48.963 INFO:teuthology.orchestra.run.smithi078.stdout:metadata: 2022-03-07T23:31:48.963 INFO:teuthology.orchestra.run.smithi078.stdout: name: psp.flannel.unprivileged 2022-03-07T23:31:48.963 INFO:teuthology.orchestra.run.smithi078.stdout: annotations: 2022-03-07T23:31:48.963 INFO:teuthology.orchestra.run.smithi078.stdout: seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default 2022-03-07T23:31:48.964 INFO:teuthology.orchestra.run.smithi078.stdout: seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default 2022-03-07T23:31:48.964 INFO:teuthology.orchestra.run.smithi078.stdout: apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default 2022-03-07T23:31:48.964 INFO:teuthology.orchestra.run.smithi078.stdout: apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default 2022-03-07T23:31:48.964 INFO:teuthology.orchestra.run.smithi078.stdout:spec: 2022-03-07T23:31:48.964 INFO:teuthology.orchestra.run.smithi078.stdout: privileged: false 2022-03-07T23:31:48.965 INFO:teuthology.orchestra.run.smithi078.stdout: volumes: 2022-03-07T23:31:48.965 INFO:teuthology.orchestra.run.smithi078.stdout: - configMap 2022-03-07T23:31:48.965 INFO:teuthology.orchestra.run.smithi078.stdout: - secret 2022-03-07T23:31:48.965 INFO:teuthology.orchestra.run.smithi078.stdout: - emptyDir 2022-03-07T23:31:48.966 INFO:teuthology.orchestra.run.smithi078.stdout: - hostPath 2022-03-07T23:31:48.966 INFO:teuthology.orchestra.run.smithi078.stdout: allowedHostPaths: 2022-03-07T23:31:48.966 INFO:teuthology.orchestra.run.smithi078.stdout: - pathPrefix: "/etc/cni/net.d" 2022-03-07T23:31:48.966 INFO:teuthology.orchestra.run.smithi078.stdout: - pathPrefix: "/etc/kube-flannel" 2022-03-07T23:31:48.967 INFO:teuthology.orchestra.run.smithi078.stdout: - pathPrefix: "/run/flannel" 2022-03-07T23:31:48.967 INFO:teuthology.orchestra.run.smithi078.stdout: readOnlyRootFilesystem: false 2022-03-07T23:31:48.967 INFO:teuthology.orchestra.run.smithi078.stdout: # Users and groups 2022-03-07T23:31:48.967 INFO:teuthology.orchestra.run.smithi078.stdout: runAsUser: 2022-03-07T23:31:48.968 INFO:teuthology.orchestra.run.smithi078.stdout: rule: RunAsAny 2022-03-07T23:31:48.968 INFO:teuthology.orchestra.run.smithi078.stdout: supplementalGroups: 2022-03-07T23:31:48.968 INFO:teuthology.orchestra.run.smithi078.stdout: rule: RunAsAny 2022-03-07T23:31:48.968 INFO:teuthology.orchestra.run.smithi078.stdout: fsGroup: 2022-03-07T23:31:48.968 INFO:teuthology.orchestra.run.smithi078.stdout: rule: RunAsAny 2022-03-07T23:31:48.969 INFO:teuthology.orchestra.run.smithi078.stdout: # Privilege Escalation 2022-03-07T23:31:48.969 INFO:teuthology.orchestra.run.smithi078.stdout: allowPrivilegeEscalation: false 2022-03-07T23:31:48.969 INFO:teuthology.orchestra.run.smithi078.stdout: defaultAllowPrivilegeEscalation: false 2022-03-07T23:31:48.969 INFO:teuthology.orchestra.run.smithi078.stdout: # Capabilities 2022-03-07T23:31:48.970 INFO:teuthology.orchestra.run.smithi078.stdout: allowedCapabilities: ['NET_ADMIN', 'NET_RAW'] 2022-03-07T23:31:48.970 INFO:teuthology.orchestra.run.smithi078.stdout: defaultAddCapabilities: [] 2022-03-07T23:31:48.970 INFO:teuthology.orchestra.run.smithi078.stdout: requiredDropCapabilities: [] 2022-03-07T23:31:48.970 INFO:teuthology.orchestra.run.smithi078.stdout: # Host namespaces 2022-03-07T23:31:48.971 INFO:teuthology.orchestra.run.smithi078.stdout: hostPID: false 2022-03-07T23:31:48.971 INFO:teuthology.orchestra.run.smithi078.stdout: hostIPC: false 2022-03-07T23:31:48.971 INFO:teuthology.orchestra.run.smithi078.stdout: hostNetwork: true 2022-03-07T23:31:48.971 INFO:teuthology.orchestra.run.smithi078.stdout: hostPorts: 2022-03-07T23:31:48.971 INFO:teuthology.orchestra.run.smithi078.stdout: - min: 0 2022-03-07T23:31:48.972 INFO:teuthology.orchestra.run.smithi078.stdout: max: 65535 2022-03-07T23:31:48.972 INFO:teuthology.orchestra.run.smithi078.stdout: # SELinux 2022-03-07T23:31:48.972 INFO:teuthology.orchestra.run.smithi078.stdout: seLinux: 2022-03-07T23:31:48.972 INFO:teuthology.orchestra.run.smithi078.stdout: # SELinux is unused in CaaSP 2022-03-07T23:31:48.973 INFO:teuthology.orchestra.run.smithi078.stdout: rule: 'RunAsAny' 2022-03-07T23:31:48.973 INFO:teuthology.orchestra.run.smithi078.stdout:--- 2022-03-07T23:31:48.973 INFO:teuthology.orchestra.run.smithi078.stdout:kind: ClusterRole 2022-03-07T23:31:48.973 INFO:teuthology.orchestra.run.smithi078.stdout:apiVersion: rbac.authorization.k8s.io/v1 2022-03-07T23:31:48.973 INFO:teuthology.orchestra.run.smithi078.stdout:metadata: 2022-03-07T23:31:48.974 INFO:teuthology.orchestra.run.smithi078.stdout: name: flannel 2022-03-07T23:31:48.974 INFO:teuthology.orchestra.run.smithi078.stdout:rules: 2022-03-07T23:31:48.974 INFO:teuthology.orchestra.run.smithi078.stdout:- apiGroups: ['extensions'] 2022-03-07T23:31:48.974 INFO:teuthology.orchestra.run.smithi078.stdout: resources: ['podsecuritypolicies'] 2022-03-07T23:31:48.975 INFO:teuthology.orchestra.run.smithi078.stdout: verbs: ['use'] 2022-03-07T23:31:48.975 INFO:teuthology.orchestra.run.smithi078.stdout: resourceNames: ['psp.flannel.unprivileged'] 2022-03-07T23:31:48.975 INFO:teuthology.orchestra.run.smithi078.stdout:- apiGroups: 2022-03-07T23:31:48.975 INFO:teuthology.orchestra.run.smithi078.stdout: - "" 2022-03-07T23:31:48.976 INFO:teuthology.orchestra.run.smithi078.stdout: resources: 2022-03-07T23:31:48.976 INFO:teuthology.orchestra.run.smithi078.stdout: - pods 2022-03-07T23:31:48.976 INFO:teuthology.orchestra.run.smithi078.stdout: verbs: 2022-03-07T23:31:48.976 INFO:teuthology.orchestra.run.smithi078.stdout: - get 2022-03-07T23:31:48.976 INFO:teuthology.orchestra.run.smithi078.stdout:- apiGroups: 2022-03-07T23:31:48.977 INFO:teuthology.orchestra.run.smithi078.stdout: - "" 2022-03-07T23:31:48.977 INFO:teuthology.orchestra.run.smithi078.stdout: resources: 2022-03-07T23:31:48.977 INFO:teuthology.orchestra.run.smithi078.stdout: - nodes 2022-03-07T23:31:48.977 INFO:teuthology.orchestra.run.smithi078.stdout: verbs: 2022-03-07T23:31:48.978 INFO:teuthology.orchestra.run.smithi078.stdout: - list 2022-03-07T23:31:48.978 INFO:teuthology.orchestra.run.smithi078.stdout: - watch 2022-03-07T23:31:48.978 INFO:teuthology.orchestra.run.smithi078.stdout:- apiGroups: 2022-03-07T23:31:48.978 INFO:teuthology.orchestra.run.smithi078.stdout: - "" 2022-03-07T23:31:48.978 INFO:teuthology.orchestra.run.smithi078.stdout: resources: 2022-03-07T23:31:48.979 INFO:teuthology.orchestra.run.smithi078.stdout: - nodes/status 2022-03-07T23:31:48.979 INFO:teuthology.orchestra.run.smithi078.stdout: verbs: 2022-03-07T23:31:48.979 INFO:teuthology.orchestra.run.smithi078.stdout: - patch 2022-03-07T23:31:48.979 INFO:teuthology.orchestra.run.smithi078.stdout:--- 2022-03-07T23:31:48.980 INFO:teuthology.orchestra.run.smithi078.stdout:kind: ClusterRoleBinding 2022-03-07T23:31:48.980 INFO:teuthology.orchestra.run.smithi078.stdout:apiVersion: rbac.authorization.k8s.io/v1 2022-03-07T23:31:48.980 INFO:teuthology.orchestra.run.smithi078.stdout:metadata: 2022-03-07T23:31:48.980 INFO:teuthology.orchestra.run.smithi078.stdout: name: flannel 2022-03-07T23:31:48.980 INFO:teuthology.orchestra.run.smithi078.stdout:roleRef: 2022-03-07T23:31:48.981 INFO:teuthology.orchestra.run.smithi078.stdout: apiGroup: rbac.authorization.k8s.io 2022-03-07T23:31:48.981 INFO:teuthology.orchestra.run.smithi078.stdout: kind: ClusterRole 2022-03-07T23:31:48.981 INFO:teuthology.orchestra.run.smithi078.stdout: name: flannel 2022-03-07T23:31:48.981 INFO:teuthology.orchestra.run.smithi078.stdout:subjects: 2022-03-07T23:31:48.982 INFO:teuthology.orchestra.run.smithi078.stdout:- kind: ServiceAccount 2022-03-07T23:31:48.982 INFO:teuthology.orchestra.run.smithi078.stdout: name: flannel 2022-03-07T23:31:48.982 INFO:teuthology.orchestra.run.smithi078.stdout: namespace: kube-system 2022-03-07T23:31:48.982 INFO:teuthology.orchestra.run.smithi078.stdout:--- 2022-03-07T23:31:48.982 INFO:teuthology.orchestra.run.smithi078.stdout:apiVersion: v1 2022-03-07T23:31:48.983 INFO:teuthology.orchestra.run.smithi078.stdout:kind: ServiceAccount 2022-03-07T23:31:48.983 INFO:teuthology.orchestra.run.smithi078.stdout:metadata: 2022-03-07T23:31:48.983 INFO:teuthology.orchestra.run.smithi078.stdout: name: flannel 2022-03-07T23:31:48.983 INFO:teuthology.orchestra.run.smithi078.stdout: namespace: kube-system 2022-03-07T23:31:48.984 INFO:teuthology.orchestra.run.smithi078.stdout:--- 2022-03-07T23:31:48.984 INFO:teuthology.orchestra.run.smithi078.stdout:kind: ConfigMap 2022-03-07T23:31:48.984 INFO:teuthology.orchestra.run.smithi078.stdout:apiVersion: v1 2022-03-07T23:31:48.984 INFO:teuthology.orchestra.run.smithi078.stdout:metadata: 2022-03-07T23:31:48.984 INFO:teuthology.orchestra.run.smithi078.stdout: name: kube-flannel-cfg 2022-03-07T23:31:48.985 INFO:teuthology.orchestra.run.smithi078.stdout: namespace: kube-system 2022-03-07T23:31:48.985 INFO:teuthology.orchestra.run.smithi078.stdout: labels: 2022-03-07T23:31:48.985 INFO:teuthology.orchestra.run.smithi078.stdout: tier: node 2022-03-07T23:31:48.985 INFO:teuthology.orchestra.run.smithi078.stdout: app: flannel 2022-03-07T23:31:48.985 INFO:teuthology.orchestra.run.smithi078.stdout:data: 2022-03-07T23:31:48.986 INFO:teuthology.orchestra.run.smithi078.stdout: cni-conf.json: | 2022-03-07T23:31:48.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2022-03-07T23:31:48.986 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cbr0", 2022-03-07T23:31:48.986 INFO:teuthology.orchestra.run.smithi078.stdout: "cniVersion": "0.3.1", 2022-03-07T23:31:48.987 INFO:teuthology.orchestra.run.smithi078.stdout: "plugins": [ 2022-03-07T23:31:48.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2022-03-07T23:31:48.987 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "flannel", 2022-03-07T23:31:48.987 INFO:teuthology.orchestra.run.smithi078.stdout: "delegate": { 2022-03-07T23:31:48.987 INFO:teuthology.orchestra.run.smithi078.stdout: "hairpinMode": true, 2022-03-07T23:31:48.988 INFO:teuthology.orchestra.run.smithi078.stdout: "isDefaultGateway": true 2022-03-07T23:31:48.988 INFO:teuthology.orchestra.run.smithi078.stdout: } 2022-03-07T23:31:48.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2022-03-07T23:31:48.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2022-03-07T23:31:48.989 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "portmap", 2022-03-07T23:31:48.989 INFO:teuthology.orchestra.run.smithi078.stdout: "capabilities": { 2022-03-07T23:31:48.989 INFO:teuthology.orchestra.run.smithi078.stdout: "portMappings": true 2022-03-07T23:31:48.989 INFO:teuthology.orchestra.run.smithi078.stdout: } 2022-03-07T23:31:48.989 INFO:teuthology.orchestra.run.smithi078.stdout: } 2022-03-07T23:31:48.990 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2022-03-07T23:31:48.990 INFO:teuthology.orchestra.run.smithi078.stdout: } 2022-03-07T23:31:48.990 INFO:teuthology.orchestra.run.smithi078.stdout: net-conf.json: | 2022-03-07T23:31:48.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2022-03-07T23:31:48.991 INFO:teuthology.orchestra.run.smithi078.stdout: "Network": "10.244.0.0/16", 2022-03-07T23:31:48.991 INFO:teuthology.orchestra.run.smithi078.stdout: "Backend": { 2022-03-07T23:31:48.991 INFO:teuthology.orchestra.run.smithi078.stdout: "Type": "vxlan" 2022-03-07T23:31:48.991 INFO:teuthology.orchestra.run.smithi078.stdout: } 2022-03-07T23:31:48.992 INFO:teuthology.orchestra.run.smithi078.stdout: } 2022-03-07T23:31:48.992 INFO:teuthology.orchestra.run.smithi078.stdout:--- 2022-03-07T23:31:48.992 INFO:teuthology.orchestra.run.smithi078.stdout:apiVersion: apps/v1 2022-03-07T23:31:48.992 INFO:teuthology.orchestra.run.smithi078.stdout:kind: DaemonSet 2022-03-07T23:31:48.992 INFO:teuthology.orchestra.run.smithi078.stdout:metadata: 2022-03-07T23:31:48.993 INFO:teuthology.orchestra.run.smithi078.stdout: name: kube-flannel-ds 2022-03-07T23:31:48.993 INFO:teuthology.orchestra.run.smithi078.stdout: namespace: kube-system 2022-03-07T23:31:48.993 INFO:teuthology.orchestra.run.smithi078.stdout: labels: 2022-03-07T23:31:48.993 INFO:teuthology.orchestra.run.smithi078.stdout: tier: node 2022-03-07T23:31:48.993 INFO:teuthology.orchestra.run.smithi078.stdout: app: flannel 2022-03-07T23:31:48.994 INFO:teuthology.orchestra.run.smithi078.stdout:spec: 2022-03-07T23:31:48.994 INFO:teuthology.orchestra.run.smithi078.stdout: selector: 2022-03-07T23:31:48.994 INFO:teuthology.orchestra.run.smithi078.stdout: matchLabels: 2022-03-07T23:31:48.994 INFO:teuthology.orchestra.run.smithi078.stdout: app: flannel 2022-03-07T23:31:48.994 INFO:teuthology.orchestra.run.smithi078.stdout: template: 2022-03-07T23:31:48.995 INFO:teuthology.orchestra.run.smithi078.stdout: metadata: 2022-03-07T23:31:48.995 INFO:teuthology.orchestra.run.smithi078.stdout: labels: 2022-03-07T23:31:48.995 INFO:teuthology.orchestra.run.smithi078.stdout: tier: node 2022-03-07T23:31:48.995 INFO:teuthology.orchestra.run.smithi078.stdout: app: flannel 2022-03-07T23:31:48.996 INFO:teuthology.orchestra.run.smithi078.stdout: spec: 2022-03-07T23:31:48.996 INFO:teuthology.orchestra.run.smithi078.stdout: affinity: 2022-03-07T23:31:48.996 INFO:teuthology.orchestra.run.smithi078.stdout: nodeAffinity: 2022-03-07T23:31:48.996 INFO:teuthology.orchestra.run.smithi078.stdout: requiredDuringSchedulingIgnoredDuringExecution: 2022-03-07T23:31:48.997 INFO:teuthology.orchestra.run.smithi078.stdout: nodeSelectorTerms: 2022-03-07T23:31:48.997 INFO:teuthology.orchestra.run.smithi078.stdout: - matchExpressions: 2022-03-07T23:31:48.997 INFO:teuthology.orchestra.run.smithi078.stdout: - key: kubernetes.io/os 2022-03-07T23:31:48.997 INFO:teuthology.orchestra.run.smithi078.stdout: operator: In 2022-03-07T23:31:48.997 INFO:teuthology.orchestra.run.smithi078.stdout: values: 2022-03-07T23:31:48.998 INFO:teuthology.orchestra.run.smithi078.stdout: - linux 2022-03-07T23:31:48.998 INFO:teuthology.orchestra.run.smithi078.stdout: hostNetwork: true 2022-03-07T23:31:48.998 INFO:teuthology.orchestra.run.smithi078.stdout: priorityClassName: system-node-critical 2022-03-07T23:31:48.998 INFO:teuthology.orchestra.run.smithi078.stdout: tolerations: 2022-03-07T23:31:48.999 INFO:teuthology.orchestra.run.smithi078.stdout: - operator: Exists 2022-03-07T23:31:48.999 INFO:teuthology.orchestra.run.smithi078.stdout: effect: NoSchedule 2022-03-07T23:31:48.999 INFO:teuthology.orchestra.run.smithi078.stdout: serviceAccountName: flannel 2022-03-07T23:31:48.999 INFO:teuthology.orchestra.run.smithi078.stdout: initContainers: 2022-03-07T23:31:49.000 INFO:teuthology.orchestra.run.smithi078.stdout: - name: install-cni-plugin 2022-03-07T23:31:49.000 INFO:teuthology.orchestra.run.smithi078.stdout: #image: flannelcni/flannel-cni-plugin:v1.0.1 for ppc64le and mips64le (dockerhub limitations may apply) 2022-03-07T23:31:49.000 INFO:teuthology.orchestra.run.smithi078.stdout: image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1 2022-03-07T23:31:49.000 INFO:teuthology.orchestra.run.smithi078.stdout: command: 2022-03-07T23:31:49.000 INFO:teuthology.orchestra.run.smithi078.stdout: - cp 2022-03-07T23:31:49.001 INFO:teuthology.orchestra.run.smithi078.stdout: args: 2022-03-07T23:31:49.001 INFO:teuthology.orchestra.run.smithi078.stdout: - -f 2022-03-07T23:31:49.001 INFO:teuthology.orchestra.run.smithi078.stdout: - /flannel 2022-03-07T23:31:49.001 INFO:teuthology.orchestra.run.smithi078.stdout: - /opt/cni/bin/flannel 2022-03-07T23:31:49.001 INFO:teuthology.orchestra.run.smithi078.stdout: volumeMounts: 2022-03-07T23:31:49.002 INFO:teuthology.orchestra.run.smithi078.stdout: - name: cni-plugin 2022-03-07T23:31:49.002 INFO:teuthology.orchestra.run.smithi078.stdout: mountPath: /opt/cni/bin 2022-03-07T23:31:49.002 INFO:teuthology.orchestra.run.smithi078.stdout: - name: install-cni 2022-03-07T23:31:49.002 INFO:teuthology.orchestra.run.smithi078.stdout: #image: flannelcni/flannel:v0.16.3 for ppc64le and mips64le (dockerhub limitations may apply) 2022-03-07T23:31:49.003 INFO:teuthology.orchestra.run.smithi078.stdout: image: rancher/mirrored-flannelcni-flannel:v0.16.3 2022-03-07T23:31:49.003 INFO:teuthology.orchestra.run.smithi078.stdout: command: 2022-03-07T23:31:49.003 INFO:teuthology.orchestra.run.smithi078.stdout: - cp 2022-03-07T23:31:49.003 INFO:teuthology.orchestra.run.smithi078.stdout: args: 2022-03-07T23:31:49.004 INFO:teuthology.orchestra.run.smithi078.stdout: - -f 2022-03-07T23:31:49.004 INFO:teuthology.orchestra.run.smithi078.stdout: - /etc/kube-flannel/cni-conf.json 2022-03-07T23:31:49.004 INFO:teuthology.orchestra.run.smithi078.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 5692 100 5692 0 0 44468 0 --:--:-- --:--:-- --:--:-- 44468 2022-03-07T23:31:49.005 INFO:teuthology.orchestra.run.smithi078.stdout: - /etc/cni/net.d/10-flannel.conflist 2022-03-07T23:31:49.006 INFO:teuthology.orchestra.run.smithi078.stdout: volumeMounts: 2022-03-07T23:31:49.006 INFO:teuthology.orchestra.run.smithi078.stdout: - name: cni 2022-03-07T23:31:49.006 INFO:teuthology.orchestra.run.smithi078.stdout: mountPath: /etc/cni/net.d 2022-03-07T23:31:49.007 INFO:teuthology.orchestra.run.smithi078.stdout: - name: flannel-cfg 2022-03-07T23:31:49.007 INFO:teuthology.orchestra.run.smithi078.stdout: mountPath: /etc/kube-flannel/ 2022-03-07T23:31:49.007 INFO:teuthology.orchestra.run.smithi078.stdout: containers: 2022-03-07T23:31:49.007 INFO:teuthology.orchestra.run.smithi078.stdout: - name: kube-flannel 2022-03-07T23:31:49.007 INFO:teuthology.orchestra.run.smithi078.stdout: #image: flannelcni/flannel:v0.16.3 for ppc64le and mips64le (dockerhub limitations may apply) 2022-03-07T23:31:49.008 INFO:teuthology.orchestra.run.smithi078.stdout: image: rancher/mirrored-flannelcni-flannel:v0.16.3 2022-03-07T23:31:49.008 INFO:teuthology.orchestra.run.smithi078.stdout: command: 2022-03-07T23:31:49.008 INFO:teuthology.orchestra.run.smithi078.stdout: - /opt/bin/flanneld 2022-03-07T23:31:49.008 INFO:teuthology.orchestra.run.smithi078.stdout: args: 2022-03-07T23:31:49.009 INFO:teuthology.orchestra.run.smithi078.stdout: - --ip-masq 2022-03-07T23:31:49.009 INFO:teuthology.orchestra.run.smithi078.stdout: - --kube-subnet-mgr 2022-03-07T23:31:49.009 INFO:teuthology.orchestra.run.smithi078.stdout: resources: 2022-03-07T23:31:49.009 INFO:teuthology.orchestra.run.smithi078.stdout: requests: 2022-03-07T23:31:49.009 INFO:teuthology.orchestra.run.smithi078.stdout: cpu: "100m" 2022-03-07T23:31:49.010 INFO:teuthology.orchestra.run.smithi078.stdout: memory: "50Mi" 2022-03-07T23:31:49.010 INFO:teuthology.orchestra.run.smithi078.stdout: limits: 2022-03-07T23:31:49.010 INFO:teuthology.orchestra.run.smithi078.stdout: cpu: "100m" 2022-03-07T23:31:49.010 INFO:teuthology.orchestra.run.smithi078.stdout: memory: "50Mi" 2022-03-07T23:31:49.011 INFO:teuthology.orchestra.run.smithi078.stdout: securityContext: 2022-03-07T23:31:49.011 INFO:teuthology.orchestra.run.smithi078.stdout: privileged: false 2022-03-07T23:31:49.011 INFO:teuthology.orchestra.run.smithi078.stdout: capabilities: 2022-03-07T23:31:49.011 INFO:teuthology.orchestra.run.smithi078.stdout: add: ["NET_ADMIN", "NET_RAW"] 2022-03-07T23:31:49.011 INFO:teuthology.orchestra.run.smithi078.stdout: env: 2022-03-07T23:31:49.012 INFO:teuthology.orchestra.run.smithi078.stdout: - name: POD_NAME 2022-03-07T23:31:49.012 INFO:teuthology.orchestra.run.smithi078.stdout: valueFrom: 2022-03-07T23:31:49.012 INFO:teuthology.orchestra.run.smithi078.stdout: fieldRef: 2022-03-07T23:31:49.012 INFO:teuthology.orchestra.run.smithi078.stdout: fieldPath: metadata.name 2022-03-07T23:31:49.013 INFO:teuthology.orchestra.run.smithi078.stdout: - name: POD_NAMESPACE 2022-03-07T23:31:49.013 INFO:teuthology.orchestra.run.smithi078.stdout: valueFrom: 2022-03-07T23:31:49.013 INFO:teuthology.orchestra.run.smithi078.stdout: fieldRef: 2022-03-07T23:31:49.013 INFO:teuthology.orchestra.run.smithi078.stdout: fieldPath: metadata.namespace 2022-03-07T23:31:49.013 INFO:teuthology.orchestra.run.smithi078.stdout: volumeMounts: 2022-03-07T23:31:49.014 INFO:teuthology.orchestra.run.smithi078.stdout: - name: run 2022-03-07T23:31:49.014 INFO:teuthology.orchestra.run.smithi078.stdout: mountPath: /run/flannel 2022-03-07T23:31:49.014 INFO:teuthology.orchestra.run.smithi078.stdout: - name: flannel-cfg 2022-03-07T23:31:49.014 INFO:teuthology.orchestra.run.smithi078.stdout: mountPath: /etc/kube-flannel/ 2022-03-07T23:31:49.014 INFO:teuthology.orchestra.run.smithi078.stdout: - name: xtables-lock 2022-03-07T23:31:49.015 INFO:teuthology.orchestra.run.smithi078.stdout: mountPath: /run/xtables.lock 2022-03-07T23:31:49.015 INFO:teuthology.orchestra.run.smithi078.stdout: volumes: 2022-03-07T23:31:49.015 INFO:teuthology.orchestra.run.smithi078.stdout: - name: run 2022-03-07T23:31:49.015 INFO:teuthology.orchestra.run.smithi078.stdout: hostPath: 2022-03-07T23:31:49.016 INFO:teuthology.orchestra.run.smithi078.stdout: path: /run/flannel 2022-03-07T23:31:49.016 INFO:teuthology.orchestra.run.smithi078.stdout: - name: cni-plugin 2022-03-07T23:31:49.016 INFO:teuthology.orchestra.run.smithi078.stdout: hostPath: 2022-03-07T23:31:49.016 INFO:teuthology.orchestra.run.smithi078.stdout: path: /opt/cni/bin 2022-03-07T23:31:49.016 INFO:teuthology.orchestra.run.smithi078.stdout: - name: cni 2022-03-07T23:31:49.017 INFO:teuthology.orchestra.run.smithi078.stdout: hostPath: 2022-03-07T23:31:49.017 INFO:teuthology.orchestra.run.smithi078.stdout: path: /etc/cni/net.d 2022-03-07T23:31:49.017 INFO:teuthology.orchestra.run.smithi078.stdout: - name: flannel-cfg 2022-03-07T23:31:49.017 INFO:teuthology.orchestra.run.smithi078.stdout: configMap: 2022-03-07T23:31:49.018 INFO:teuthology.orchestra.run.smithi078.stdout: name: kube-flannel-cfg 2022-03-07T23:31:49.018 INFO:teuthology.orchestra.run.smithi078.stdout: - name: xtables-lock 2022-03-07T23:31:49.018 INFO:teuthology.orchestra.run.smithi078.stdout: hostPath: 2022-03-07T23:31:49.018 INFO:teuthology.orchestra.run.smithi078.stdout: path: /run/xtables.lock 2022-03-07T23:31:49.019 INFO:teuthology.orchestra.run.smithi078.stdout: type: FileOrCreate 2022-03-07T23:31:49.046 INFO:tasks.kubeadm:Updating {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'kube-flannel-cfg', 'namespace': 'kube-system', 'labels': {'tier': 'node', 'app': 'flannel'}}, 'data': {'cni-conf.json': '{\n "name": "cbr0",\n "cniVersion": "0.3.1",\n "plugins": [\n {\n "type": "flannel",\n "delegate": {\n "hairpinMode": true,\n "isDefaultGateway": true\n }\n },\n {\n "type": "portmap",\n "capabilities": {\n "portMappings": true\n }\n }\n ]\n}\n', 'net-conf.json': '{\n "Network": "10.244.0.0/16",\n "Backend": {\n "Type": "vxlan"\n }\n}\n'}} 2022-03-07T23:31:49.047 INFO:tasks.kubeadm:Now {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'kube-flannel-cfg', 'namespace': 'kube-system', 'labels': {'tier': 'node', 'app': 'flannel'}}, 'data': {'cni-conf.json': '{\n "name": "cbr0",\n "cniVersion": "0.3.1",\n "plugins": [\n {\n "type": "flannel",\n "delegate": {\n "hairpinMode": true,\n "isDefaultGateway": true\n }\n },\n {\n "type": "portmap",\n "capabilities": {\n "portMappings": true\n }\n }\n ]\n}\n', 'net-conf.json': '{\n "Network": "10.250.104.0/21",\n "Backend": {\n "Type": "vxlan"\n }\n}\n'}} 2022-03-07T23:31:49.061 DEBUG:tasks.kubeadm:Flannel: apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: annotations: apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default name: psp.flannel.unprivileged spec: allowPrivilegeEscalation: false allowedCapabilities: - NET_ADMIN - NET_RAW allowedHostPaths: - pathPrefix: /etc/cni/net.d - pathPrefix: /etc/kube-flannel - pathPrefix: /run/flannel defaultAddCapabilities: [] defaultAllowPrivilegeEscalation: false fsGroup: rule: RunAsAny hostIPC: false hostNetwork: true hostPID: false hostPorts: - max: 65535 min: 0 privileged: false readOnlyRootFilesystem: false requiredDropCapabilities: [] runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny volumes: - configMap - secret - emptyDir - hostPath --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: flannel rules: - apiGroups: - extensions resourceNames: - psp.flannel.unprivileged resources: - podsecuritypolicies verbs: - use - apiGroups: - '' resources: - pods verbs: - get - apiGroups: - '' resources: - nodes verbs: - list - watch - apiGroups: - '' resources: - nodes/status verbs: - patch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: flannel roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannel subjects: - kind: ServiceAccount name: flannel namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: flannel namespace: kube-system --- apiVersion: v1 data: cni-conf.json: "{\n \"name\": \"cbr0\",\n \"cniVersion\": \"0.3.1\",\n \"plugins\"\ : [\n {\n \"type\": \"flannel\",\n \"delegate\": {\n \"hairpinMode\"\ : true,\n \"isDefaultGateway\": true\n }\n },\n {\n \"\ type\": \"portmap\",\n \"capabilities\": {\n \"portMappings\": true\n\ \ }\n }\n ]\n}\n" net-conf.json: "{\n \"Network\": \"10.250.104.0/21\",\n \"Backend\": {\n \"\ Type\": \"vxlan\"\n }\n}\n" kind: ConfigMap metadata: labels: app: flannel tier: node name: kube-flannel-cfg namespace: kube-system --- apiVersion: apps/v1 kind: DaemonSet metadata: labels: app: flannel tier: node name: kube-flannel-ds namespace: kube-system spec: selector: matchLabels: app: flannel template: metadata: labels: app: flannel tier: node spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - linux containers: - args: - --ip-masq - --kube-subnet-mgr command: - /opt/bin/flanneld env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: rancher/mirrored-flannelcni-flannel:v0.16.3 name: kube-flannel resources: limits: cpu: 100m memory: 50Mi requests: cpu: 100m memory: 50Mi securityContext: capabilities: add: - NET_ADMIN - NET_RAW privileged: false volumeMounts: - mountPath: /run/flannel name: run - mountPath: /etc/kube-flannel/ name: flannel-cfg - mountPath: /run/xtables.lock name: xtables-lock hostNetwork: true initContainers: - args: - -f - /flannel - /opt/cni/bin/flannel command: - cp image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1 name: install-cni-plugin volumeMounts: - mountPath: /opt/cni/bin name: cni-plugin - args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist command: - cp image: rancher/mirrored-flannelcni-flannel:v0.16.3 name: install-cni volumeMounts: - mountPath: /etc/cni/net.d name: cni - mountPath: /etc/kube-flannel/ name: flannel-cfg priorityClassName: system-node-critical serviceAccountName: flannel tolerations: - effect: NoSchedule operator: Exists volumes: - hostPath: path: /run/flannel name: run - hostPath: path: /opt/cni/bin name: cni-plugin - hostPath: path: /etc/cni/net.d name: cni - configMap: name: kube-flannel-cfg name: flannel-cfg - hostPath: path: /run/xtables.lock type: FileOrCreate name: xtables-lock 2022-03-07T23:31:49.061 DEBUG:teuthology.orchestra.run.smithi078:> kubectl apply -f - 2022-03-07T23:31:51.376 INFO:teuthology.orchestra.run.smithi078.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-07T23:31:51.384 INFO:teuthology.orchestra.run.smithi078.stdout:podsecuritypolicy.policy/psp.flannel.unprivileged created 2022-03-07T23:31:51.416 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrole.rbac.authorization.k8s.io/flannel created 2022-03-07T23:31:51.429 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrolebinding.rbac.authorization.k8s.io/flannel created 2022-03-07T23:31:51.473 INFO:teuthology.orchestra.run.smithi078.stdout:serviceaccount/flannel created 2022-03-07T23:31:51.490 INFO:teuthology.orchestra.run.smithi078.stdout:configmap/kube-flannel-cfg created 2022-03-07T23:31:51.500 INFO:teuthology.orchestra.run.smithi078.stdout:daemonset.apps/kube-flannel-ds created 2022-03-07T23:31:51.504 INFO:tasks.kubeadm:Scanning for scratch devices 2022-03-07T23:31:51.504 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2022-03-07T23:31:51.505 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/scratch_devs of=/dev/stdout 2022-03-07T23:31:51.511 INFO:tasks.kubeadm:Scratch devices on smithi078: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-07T23:31:51.511 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-07T23:31:51.538 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records in 2022-03-07T23:31:51.540 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2022-03-07T23:31:51.540 INFO:teuthology.orchestra.run.smithi078.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0134034 s, 782 MB/s 2022-03-07T23:31:51.541 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-07T23:31:51.609 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records in 2022-03-07T23:31:51.609 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2022-03-07T23:31:51.610 INFO:teuthology.orchestra.run.smithi078.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0155928 s, 672 MB/s 2022-03-07T23:31:51.612 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-07T23:31:51.637 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records in 2022-03-07T23:31:51.638 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2022-03-07T23:31:51.638 INFO:teuthology.orchestra.run.smithi078.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0151709 s, 691 MB/s 2022-03-07T23:31:51.639 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-07T23:31:51.664 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records in 2022-03-07T23:31:51.664 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2022-03-07T23:31:51.665 INFO:teuthology.orchestra.run.smithi078.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0140805 s, 745 MB/s 2022-03-07T23:31:51.666 DEBUG:teuthology.orchestra.run.smithi166:> set -ex 2022-03-07T23:31:51.666 DEBUG:teuthology.orchestra.run.smithi166:> dd if=/scratch_devs of=/dev/stdout 2022-03-07T23:31:51.672 INFO:tasks.kubeadm:Scratch devices on smithi166: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-07T23:31:51.673 DEBUG:teuthology.orchestra.run.smithi166:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-07T23:31:51.743 INFO:teuthology.orchestra.run.smithi166.stderr:10+0 records in 2022-03-07T23:31:51.743 INFO:teuthology.orchestra.run.smithi166.stderr:10+0 records out 2022-03-07T23:31:51.744 INFO:teuthology.orchestra.run.smithi166.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0167406 s, 626 MB/s 2022-03-07T23:31:51.745 DEBUG:teuthology.orchestra.run.smithi166:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-07T23:31:51.775 INFO:teuthology.orchestra.run.smithi166.stderr:10+0 records in 2022-03-07T23:31:51.775 INFO:teuthology.orchestra.run.smithi166.stderr:10+0 records out 2022-03-07T23:31:51.775 INFO:teuthology.orchestra.run.smithi166.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0161405 s, 650 MB/s 2022-03-07T23:31:51.777 DEBUG:teuthology.orchestra.run.smithi166:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-07T23:31:51.806 INFO:teuthology.orchestra.run.smithi166.stderr:10+0 records in 2022-03-07T23:31:51.806 INFO:teuthology.orchestra.run.smithi166.stderr:10+0 records out 2022-03-07T23:31:51.807 INFO:teuthology.orchestra.run.smithi166.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.014993 s, 699 MB/s 2022-03-07T23:31:51.808 DEBUG:teuthology.orchestra.run.smithi166:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-07T23:31:51.879 INFO:teuthology.orchestra.run.smithi166.stderr:10+0 records in 2022-03-07T23:31:51.879 INFO:teuthology.orchestra.run.smithi166.stderr:10+0 records out 2022-03-07T23:31:51.880 INFO:teuthology.orchestra.run.smithi166.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0169186 s, 620 MB/s 2022-03-07T23:31:51.881 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2022-03-07T23:31:51.881 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/scratch_devs of=/dev/stdout 2022-03-07T23:31:51.889 INFO:tasks.kubeadm:Scratch devices on smithi170: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-07T23:31:51.890 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-03-07T23:31:51.916 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records in 2022-03-07T23:31:51.917 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2022-03-07T23:31:51.917 INFO:teuthology.orchestra.run.smithi170.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0155799 s, 673 MB/s 2022-03-07T23:31:51.918 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-03-07T23:31:51.993 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records in 2022-03-07T23:31:51.994 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2022-03-07T23:31:51.994 INFO:teuthology.orchestra.run.smithi170.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0141788 s, 740 MB/s 2022-03-07T23:31:51.995 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-03-07T23:31:52.023 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records in 2022-03-07T23:31:52.023 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2022-03-07T23:31:52.023 INFO:teuthology.orchestra.run.smithi170.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0147271 s, 712 MB/s 2022-03-07T23:31:52.025 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-03-07T23:31:52.047 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records in 2022-03-07T23:31:52.048 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2022-03-07T23:31:52.048 INFO:teuthology.orchestra.run.smithi170.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124559 s, 842 MB/s 2022-03-07T23:31:52.074 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-03-07T23:31:52.074 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi078-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi078 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi078-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi078 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi078-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi078 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi078-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi078 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi166-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi166 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi166-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi166 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi166-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi166 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi166-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi166 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi170-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi170 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi170-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi170 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi170-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi170 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi170-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi170 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-03-07T23:31:52.075 DEBUG:teuthology.orchestra.run.smithi078:> kubectl create -f - 2022-03-07T23:31:52.541 INFO:teuthology.orchestra.run.smithi078.stdout:persistentvolume/smithi078-lv-1 created 2022-03-07T23:31:52.713 INFO:teuthology.orchestra.run.smithi078.stdout:persistentvolume/smithi078-lv-2 created 2022-03-07T23:31:52.716 INFO:teuthology.orchestra.run.smithi078.stdout:persistentvolume/smithi078-lv-3 created 2022-03-07T23:31:52.781 INFO:teuthology.orchestra.run.smithi078.stdout:persistentvolume/smithi078-lv-4 created 2022-03-07T23:31:52.880 INFO:teuthology.orchestra.run.smithi078.stdout:persistentvolume/smithi166-lv-1 created 2022-03-07T23:31:52.884 INFO:teuthology.orchestra.run.smithi078.stdout:persistentvolume/smithi166-lv-2 created 2022-03-07T23:31:52.920 INFO:teuthology.orchestra.run.smithi078.stdout:persistentvolume/smithi166-lv-3 created 2022-03-07T23:31:52.962 INFO:teuthology.orchestra.run.smithi078.stdout:persistentvolume/smithi166-lv-4 created 2022-03-07T23:31:52.968 INFO:teuthology.orchestra.run.smithi078.stdout:persistentvolume/smithi170-lv-1 created 2022-03-07T23:31:53.080 INFO:teuthology.orchestra.run.smithi078.stdout:persistentvolume/smithi170-lv-2 created 2022-03-07T23:31:53.082 INFO:teuthology.orchestra.run.smithi078.stdout:persistentvolume/smithi170-lv-3 created 2022-03-07T23:31:53.121 INFO:teuthology.orchestra.run.smithi078.stdout:persistentvolume/smithi170-lv-4 created 2022-03-07T23:31:53.154 INFO:teuthology.orchestra.run.smithi078.stdout:storageclass.storage.k8s.io/scratch created 2022-03-07T23:31:53.158 DEBUG:teuthology.orchestra.run.smithi078:> kubectl taint node smithi078 node-role.kubernetes.io/master- || true 2022-03-07T23:31:53.256 INFO:teuthology.orchestra.run.smithi078.stdout:node/smithi078 untainted 2022-03-07T23:31:53.259 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-03-07T23:31:53.260 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-03-07T23:31:53.274 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-03-07T23:31:53.275 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2022-03-07T23:31:53.276 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/scratch_devs of=/dev/stdout 2022-03-07T23:31:53.283 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-07T23:31:53.284 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-07T23:31:53.331 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-07T23:31:53.332 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-07T23:31:53.332 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 499 Links: 1 2022-03-07T23:31:53.332 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-07T23:31:53.332 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2022-03-07 23:31:51.542603762 +0000 2022-03-07T23:31:53.333 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2022-03-07 23:31:51.542603762 +0000 2022-03-07T23:31:53.333 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2022-03-07 23:31:51.542603762 +0000 2022-03-07T23:31:53.333 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2022-03-07T23:31:53.340 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2022-03-07T23:31:53.340 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2022-03-07T23:31:53.341 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.00059007 s, 868 kB/s 2022-03-07T23:31:53.344 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-07T23:31:53.348 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-07T23:31:53.349 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-07T23:31:53.349 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 508 Links: 1 2022-03-07T23:31:53.349 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-07T23:31:53.349 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2022-03-07 23:31:51.614603760 +0000 2022-03-07T23:31:53.350 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2022-03-07 23:31:51.614603760 +0000 2022-03-07T23:31:53.350 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2022-03-07 23:31:51.614603760 +0000 2022-03-07T23:31:53.350 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2022-03-07T23:31:53.355 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2022-03-07T23:31:53.356 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2022-03-07T23:31:53.356 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000539581 s, 949 kB/s 2022-03-07T23:31:53.359 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-07T23:31:53.364 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-07T23:31:53.364 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-07T23:31:53.365 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 511 Links: 1 2022-03-07T23:31:53.365 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-07T23:31:53.365 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2022-03-07 23:31:51.646603760 +0000 2022-03-07T23:31:53.365 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2022-03-07 23:31:51.642603760 +0000 2022-03-07T23:31:53.366 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2022-03-07 23:31:51.642603760 +0000 2022-03-07T23:31:53.366 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2022-03-07T23:31:53.371 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2022-03-07T23:31:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2022-03-07T23:31:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000471735 s, 1.1 MB/s 2022-03-07T23:31:53.376 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-07T23:31:53.382 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-07T23:31:53.382 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-07T23:31:53.382 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 519 Links: 1 2022-03-07T23:31:53.383 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-07T23:31:53.383 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2022-03-07 23:31:51.666603759 +0000 2022-03-07T23:31:53.383 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2022-03-07 23:31:51.666603759 +0000 2022-03-07T23:31:53.383 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2022-03-07 23:31:51.666603759 +0000 2022-03-07T23:31:53.384 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2022-03-07T23:31:53.388 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2022-03-07T23:31:53.389 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2022-03-07T23:31:53.389 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000555521 s, 922 kB/s 2022-03-07T23:31:53.392 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-07T23:31:53.481 INFO:teuthology.orchestra.run.smithi078.stdout:loop 2022-03-07T23:31:53.483 INFO:tasks.nvme_loop:Connecting nvme_loop smithi078:/dev/vg_nvme/lv_1... 2022-03-07T23:31:53.483 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-07T23:31:53.501 INFO:teuthology.orchestra.run.smithi078.stdout:1 2022-03-07T23:31:53.519 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/vg_nvme/lv_11 2022-03-07T23:31:53.541 INFO:tasks.nvme_loop:Connecting nvme_loop smithi078:/dev/vg_nvme/lv_2... 2022-03-07T23:31:53.541 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-07T23:31:53.556 INFO:teuthology.orchestra.run.smithi078.stdout:1 2022-03-07T23:31:53.583 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/vg_nvme/lv_21 2022-03-07T23:31:53.599 INFO:tasks.nvme_loop:Connecting nvme_loop smithi078:/dev/vg_nvme/lv_3... 2022-03-07T23:31:53.599 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-07T23:31:53.665 INFO:teuthology.orchestra.run.smithi078.stdout:1 2022-03-07T23:31:53.696 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/vg_nvme/lv_31 2022-03-07T23:31:53.715 INFO:tasks.nvme_loop:Connecting nvme_loop smithi078:/dev/vg_nvme/lv_4... 2022-03-07T23:31:53.715 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-07T23:31:53.735 INFO:teuthology.orchestra.run.smithi078.stdout:1 2022-03-07T23:31:53.756 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/vg_nvme/lv_41 2022-03-07T23:31:53.775 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2022-03-07T23:31:53.775 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/scratch_devs of=/dev/stdout 2022-03-07T23:31:53.825 DEBUG:teuthology.orchestra.run.smithi078:> sudo nvme list 2022-03-07T23:31:53.840 INFO:teuthology.orchestra.run.smithi078.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-07T23:31:53.841 INFO:teuthology.orchestra.run.smithi078.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-07T23:31:53.841 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/nvme0n1 PHFT620400QR400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-07T23:31:53.841 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/nvme1n1 4e52ffe3726166cb Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T23:31:53.841 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/nvme2n1 cc7480f1c4df7aee Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T23:31:53.842 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/nvme3n1 4c7cb438d7e79226 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T23:31:53.842 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/nvme4n1 6e5549a2632133a5 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T23:31:53.843 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-07T23:31:53.843 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2022-03-07T23:31:53.844 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/scratch_devs 2022-03-07T23:31:53.858 DEBUG:teuthology.orchestra.run.smithi166:> set -ex 2022-03-07T23:31:53.858 DEBUG:teuthology.orchestra.run.smithi166:> dd if=/scratch_devs of=/dev/stdout 2022-03-07T23:31:53.866 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-07T23:31:53.867 DEBUG:teuthology.orchestra.run.smithi166:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-07T23:31:53.916 INFO:teuthology.orchestra.run.smithi166.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-07T23:31:53.916 INFO:teuthology.orchestra.run.smithi166.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-07T23:31:53.916 INFO:teuthology.orchestra.run.smithi166.stdout:Device: 5h/5d Inode: 500 Links: 1 2022-03-07T23:31:53.917 INFO:teuthology.orchestra.run.smithi166.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-07T23:31:53.917 INFO:teuthology.orchestra.run.smithi166.stdout:Access: 2022-03-07 23:31:51.744691617 +0000 2022-03-07T23:31:53.917 INFO:teuthology.orchestra.run.smithi166.stdout:Modify: 2022-03-07 23:31:51.744691617 +0000 2022-03-07T23:31:53.917 INFO:teuthology.orchestra.run.smithi166.stdout:Change: 2022-03-07 23:31:51.744691617 +0000 2022-03-07T23:31:53.918 INFO:teuthology.orchestra.run.smithi166.stdout: Birth: - 2022-03-07T23:31:53.926 INFO:teuthology.orchestra.run.smithi166.stderr:1+0 records in 2022-03-07T23:31:53.926 INFO:teuthology.orchestra.run.smithi166.stderr:1+0 records out 2022-03-07T23:31:53.927 INFO:teuthology.orchestra.run.smithi166.stderr:512 bytes copied, 0.000558608 s, 917 kB/s 2022-03-07T23:31:53.931 DEBUG:teuthology.orchestra.run.smithi166:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-07T23:31:53.938 INFO:teuthology.orchestra.run.smithi166.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-07T23:31:53.938 INFO:teuthology.orchestra.run.smithi166.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-07T23:31:53.939 INFO:teuthology.orchestra.run.smithi166.stdout:Device: 5h/5d Inode: 508 Links: 1 2022-03-07T23:31:53.939 INFO:teuthology.orchestra.run.smithi166.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-07T23:31:53.939 INFO:teuthology.orchestra.run.smithi166.stdout:Access: 2022-03-07 23:31:51.776691616 +0000 2022-03-07T23:31:53.939 INFO:teuthology.orchestra.run.smithi166.stdout:Modify: 2022-03-07 23:31:51.776691616 +0000 2022-03-07T23:31:53.939 INFO:teuthology.orchestra.run.smithi166.stdout:Change: 2022-03-07 23:31:51.776691616 +0000 2022-03-07T23:31:53.940 INFO:teuthology.orchestra.run.smithi166.stdout: Birth: - 2022-03-07T23:31:53.944 INFO:teuthology.orchestra.run.smithi166.stderr:1+0 records in 2022-03-07T23:31:53.945 INFO:teuthology.orchestra.run.smithi166.stderr:1+0 records out 2022-03-07T23:31:53.945 INFO:teuthology.orchestra.run.smithi166.stderr:512 bytes copied, 0.000482637 s, 1.1 MB/s 2022-03-07T23:31:53.949 DEBUG:teuthology.orchestra.run.smithi166:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-07T23:31:53.955 INFO:teuthology.orchestra.run.smithi166.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-07T23:31:53.955 INFO:teuthology.orchestra.run.smithi166.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-07T23:31:53.955 INFO:teuthology.orchestra.run.smithi166.stdout:Device: 5h/5d Inode: 513 Links: 1 2022-03-07T23:31:53.956 INFO:teuthology.orchestra.run.smithi166.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-07T23:31:53.956 INFO:teuthology.orchestra.run.smithi166.stdout:Access: 2022-03-07 23:31:51.808691615 +0000 2022-03-07T23:31:53.956 INFO:teuthology.orchestra.run.smithi166.stdout:Modify: 2022-03-07 23:31:51.808691615 +0000 2022-03-07T23:31:53.956 INFO:teuthology.orchestra.run.smithi166.stdout:Change: 2022-03-07 23:31:51.808691615 +0000 2022-03-07T23:31:53.957 INFO:teuthology.orchestra.run.smithi166.stdout: Birth: - 2022-03-07T23:31:53.962 INFO:teuthology.orchestra.run.smithi166.stderr:1+0 records in 2022-03-07T23:31:53.962 INFO:teuthology.orchestra.run.smithi166.stderr:1+0 records out 2022-03-07T23:31:53.963 INFO:teuthology.orchestra.run.smithi166.stderr:512 bytes copied, 0.000483438 s, 1.1 MB/s 2022-03-07T23:31:53.967 DEBUG:teuthology.orchestra.run.smithi166:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-07T23:31:53.973 INFO:teuthology.orchestra.run.smithi166.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-07T23:31:53.973 INFO:teuthology.orchestra.run.smithi166.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-07T23:31:53.974 INFO:teuthology.orchestra.run.smithi166.stdout:Device: 5h/5d Inode: 518 Links: 1 2022-03-07T23:31:53.974 INFO:teuthology.orchestra.run.smithi166.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-07T23:31:53.974 INFO:teuthology.orchestra.run.smithi166.stdout:Access: 2022-03-07 23:31:51.884691614 +0000 2022-03-07T23:31:53.974 INFO:teuthology.orchestra.run.smithi166.stdout:Modify: 2022-03-07 23:31:51.884691614 +0000 2022-03-07T23:31:53.975 INFO:teuthology.orchestra.run.smithi166.stdout:Change: 2022-03-07 23:31:51.884691614 +0000 2022-03-07T23:31:53.975 INFO:teuthology.orchestra.run.smithi166.stdout: Birth: - 2022-03-07T23:31:53.982 INFO:teuthology.orchestra.run.smithi166.stderr:1+0 records in 2022-03-07T23:31:53.983 INFO:teuthology.orchestra.run.smithi166.stderr:1+0 records out 2022-03-07T23:31:53.983 INFO:teuthology.orchestra.run.smithi166.stderr:512 bytes copied, 0.000468119 s, 1.1 MB/s 2022-03-07T23:31:53.987 DEBUG:teuthology.orchestra.run.smithi166:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-07T23:31:54.159 INFO:teuthology.orchestra.run.smithi166.stdout:loop 2022-03-07T23:31:54.161 INFO:tasks.nvme_loop:Connecting nvme_loop smithi166:/dev/vg_nvme/lv_1... 2022-03-07T23:31:54.161 DEBUG:teuthology.orchestra.run.smithi166:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-07T23:31:54.185 INFO:teuthology.orchestra.run.smithi166.stdout:1 2022-03-07T23:31:54.209 INFO:teuthology.orchestra.run.smithi166.stdout:/dev/vg_nvme/lv_11 2022-03-07T23:31:54.231 INFO:tasks.nvme_loop:Connecting nvme_loop smithi166:/dev/vg_nvme/lv_2... 2022-03-07T23:31:54.231 DEBUG:teuthology.orchestra.run.smithi166:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-07T23:31:54.254 INFO:teuthology.orchestra.run.smithi166.stdout:1 2022-03-07T23:31:54.276 INFO:teuthology.orchestra.run.smithi166.stdout:/dev/vg_nvme/lv_21 2022-03-07T23:31:54.295 INFO:tasks.nvme_loop:Connecting nvme_loop smithi166:/dev/vg_nvme/lv_3... 2022-03-07T23:31:54.297 DEBUG:teuthology.orchestra.run.smithi166:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-07T23:31:54.314 INFO:teuthology.orchestra.run.smithi166.stdout:1 2022-03-07T23:31:54.335 INFO:teuthology.orchestra.run.smithi166.stdout:/dev/vg_nvme/lv_31 2022-03-07T23:31:54.350 INFO:tasks.nvme_loop:Connecting nvme_loop smithi166:/dev/vg_nvme/lv_4... 2022-03-07T23:31:54.350 DEBUG:teuthology.orchestra.run.smithi166:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-07T23:31:54.415 INFO:teuthology.orchestra.run.smithi166.stdout:1 2022-03-07T23:31:54.434 INFO:teuthology.orchestra.run.smithi166.stdout:/dev/vg_nvme/lv_41 2022-03-07T23:31:54.453 DEBUG:teuthology.orchestra.run.smithi166:> set -ex 2022-03-07T23:31:54.454 DEBUG:teuthology.orchestra.run.smithi166:> dd if=/scratch_devs of=/dev/stdout 2022-03-07T23:31:54.460 DEBUG:teuthology.orchestra.run.smithi166:> sudo nvme list 2022-03-07T23:31:54.470 INFO:teuthology.orchestra.run.smithi166.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-07T23:31:54.471 INFO:teuthology.orchestra.run.smithi166.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-07T23:31:54.471 INFO:teuthology.orchestra.run.smithi166.stdout:/dev/nvme0n1 PHFT620401AC400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-03-07T23:31:54.471 INFO:teuthology.orchestra.run.smithi166.stdout:/dev/nvme1n1 0b5fb39dce3aa718 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T23:31:54.472 INFO:teuthology.orchestra.run.smithi166.stdout:/dev/nvme2n1 ab55d18da8568aa5 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T23:31:54.472 INFO:teuthology.orchestra.run.smithi166.stdout:/dev/nvme3n1 2b0902bc67a8c395 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T23:31:54.472 INFO:teuthology.orchestra.run.smithi166.stdout:/dev/nvme4n1 03c8b63d6a477add Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T23:31:54.473 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-07T23:31:54.473 DEBUG:teuthology.orchestra.run.smithi166:> set -ex 2022-03-07T23:31:54.474 DEBUG:teuthology.orchestra.run.smithi166:> sudo dd of=/scratch_devs 2022-03-07T23:31:54.535 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2022-03-07T23:31:54.536 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/scratch_devs of=/dev/stdout 2022-03-07T23:31:54.544 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-03-07T23:31:54.544 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-03-07T23:31:54.550 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-03-07T23:31:54.550 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-07T23:31:54.551 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 500 Links: 1 2022-03-07T23:31:54.551 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-07T23:31:54.551 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2022-03-07 23:31:51.921355411 +0000 2022-03-07T23:31:54.551 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2022-03-07 23:31:51.917355411 +0000 2022-03-07T23:31:54.552 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2022-03-07 23:31:51.917355411 +0000 2022-03-07T23:31:54.552 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: - 2022-03-07T23:31:54.561 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2022-03-07T23:31:54.561 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2022-03-07T23:31:54.562 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000319668 s, 1.6 MB/s 2022-03-07T23:31:54.566 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-03-07T23:31:54.572 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-03-07T23:31:54.572 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-07T23:31:54.573 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 506 Links: 1 2022-03-07T23:31:54.573 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-07T23:31:54.573 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2022-03-07 23:31:51.997355409 +0000 2022-03-07T23:31:54.573 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2022-03-07 23:31:51.993355410 +0000 2022-03-07T23:31:54.574 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2022-03-07 23:31:51.993355410 +0000 2022-03-07T23:31:54.574 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: - 2022-03-07T23:31:54.582 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2022-03-07T23:31:54.582 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2022-03-07T23:31:54.583 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.00025899 s, 2.0 MB/s 2022-03-07T23:31:54.587 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-03-07T23:31:54.635 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-03-07T23:31:54.636 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-07T23:31:54.636 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 511 Links: 1 2022-03-07T23:31:54.636 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-07T23:31:54.637 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2022-03-07 23:31:52.025355409 +0000 2022-03-07T23:31:54.637 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2022-03-07 23:31:52.025355409 +0000 2022-03-07T23:31:54.637 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2022-03-07 23:31:52.025355409 +0000 2022-03-07T23:31:54.637 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: - 2022-03-07T23:31:54.645 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2022-03-07T23:31:54.645 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2022-03-07T23:31:54.645 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000267974 s, 1.9 MB/s 2022-03-07T23:31:54.648 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-03-07T23:31:54.654 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-03-07T23:31:54.654 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-03-07T23:31:54.655 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 519 Links: 1 2022-03-07T23:31:54.655 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-03-07T23:31:54.655 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2022-03-07 23:31:52.053355408 +0000 2022-03-07T23:31:54.655 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2022-03-07 23:31:52.053355408 +0000 2022-03-07T23:31:54.656 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2022-03-07 23:31:52.053355408 +0000 2022-03-07T23:31:54.656 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: - 2022-03-07T23:31:54.664 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2022-03-07T23:31:54.664 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2022-03-07T23:31:54.664 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000341824 s, 1.5 MB/s 2022-03-07T23:31:54.667 DEBUG:teuthology.orchestra.run.smithi170:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-03-07T23:31:54.775 INFO:teuthology.orchestra.run.smithi170.stdout:loop 2022-03-07T23:31:54.777 INFO:tasks.nvme_loop:Connecting nvme_loop smithi170:/dev/vg_nvme/lv_1... 2022-03-07T23:31:54.777 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-03-07T23:31:54.801 INFO:teuthology.orchestra.run.smithi170.stdout:1 2022-03-07T23:31:54.826 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/vg_nvme/lv_11 2022-03-07T23:31:54.849 INFO:tasks.nvme_loop:Connecting nvme_loop smithi170:/dev/vg_nvme/lv_2... 2022-03-07T23:31:54.849 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-03-07T23:31:54.913 INFO:teuthology.orchestra.run.smithi170.stdout:1 2022-03-07T23:31:54.937 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/vg_nvme/lv_21 2022-03-07T23:31:54.960 INFO:tasks.nvme_loop:Connecting nvme_loop smithi170:/dev/vg_nvme/lv_3... 2022-03-07T23:31:54.961 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-03-07T23:31:54.986 INFO:teuthology.orchestra.run.smithi170.stdout:1 2022-03-07T23:31:55.009 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/vg_nvme/lv_31 2022-03-07T23:31:55.031 INFO:tasks.nvme_loop:Connecting nvme_loop smithi170:/dev/vg_nvme/lv_4... 2022-03-07T23:31:55.033 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-03-07T23:31:55.094 INFO:teuthology.orchestra.run.smithi170.stdout:1 2022-03-07T23:31:55.114 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/vg_nvme/lv_41 2022-03-07T23:31:55.136 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2022-03-07T23:31:55.136 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/scratch_devs of=/dev/stdout 2022-03-07T23:31:55.143 DEBUG:teuthology.orchestra.run.smithi170:> sudo nvme list 2022-03-07T23:31:55.154 INFO:teuthology.orchestra.run.smithi170.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-07T23:31:55.155 INFO:teuthology.orchestra.run.smithi170.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-07T23:31:55.155 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/nvme0n1 PHMB7476002R480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2022-03-07T23:31:55.155 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/nvme1n1 dcc276c51d853cea Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T23:31:55.155 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/nvme2n1 5c7e7b3737f4269d Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T23:31:55.156 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/nvme3n1 da544f0db8398100 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T23:31:55.157 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1'] 2022-03-07T23:31:56.158 DEBUG:teuthology.orchestra.run.smithi170:> sudo nvme list 2022-03-07T23:31:56.168 INFO:teuthology.orchestra.run.smithi170.stdout:Node SN Model Namespace Usage Format FW Rev 2022-03-07T23:31:56.168 INFO:teuthology.orchestra.run.smithi170.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-03-07T23:31:56.168 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/nvme0n1 PHMB7476002R480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2022-03-07T23:31:56.169 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/nvme1n1 dcc276c51d853cea Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T23:31:56.169 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/nvme2n1 5c7e7b3737f4269d Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T23:31:56.169 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/nvme3n1 da544f0db8398100 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T23:31:56.169 INFO:teuthology.orchestra.run.smithi170.stdout:/dev/nvme4n1 815252e8c02b7698 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.13.0-3 2022-03-07T23:31:56.170 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-03-07T23:31:56.171 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2022-03-07T23:31:56.171 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/scratch_devs 2022-03-07T23:31:56.338 INFO:teuthology.run_tasks:Running task rook... 2022-03-07T23:31:56.411 INFO:tasks.rook:Rook start 2022-03-07T23:31:56.412 INFO:tasks.rook:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '94cb3430b74fb099b8927e3e6ad50dfd969173f4', 'rook_image': 'rook/ceph:master'} 2022-03-07T23:31:56.413 INFO:tasks.rook:Ceph image is quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4 2022-03-07T23:31:56.413 INFO:tasks.rook:Cloning https://github.com/rook/rook branch master 2022-03-07T23:31:56.413 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf rook && git clone --single-branch --branch master https://github.com/rook/rook rook 2022-03-07T23:31:56.421 INFO:teuthology.orchestra.run.smithi078.stderr:Cloning into 'rook'... 2022-03-07T23:31:58.912 INFO:tasks.rook:/home/teuthworker 2022-03-07T23:31:58.913 DEBUG:teuthology.orchestra.run.smithi078:> PATH=/sbin:/usr/sbin ip addr show 2022-03-07T23:31:58.920 INFO:teuthology.orchestra.run.smithi078.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-03-07T23:31:58.920 INFO:teuthology.orchestra.run.smithi078.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-03-07T23:31:58.920 INFO:teuthology.orchestra.run.smithi078.stdout: inet 127.0.0.1/8 scope host lo 2022-03-07T23:31:58.921 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2022-03-07T23:31:58.921 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 ::1/128 scope host 2022-03-07T23:31:58.921 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2022-03-07T23:31:58.921 INFO:teuthology.orchestra.run.smithi078.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-07T23:31:58.922 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:22 brd ff:ff:ff:ff:ff:ff 2022-03-07T23:31:58.922 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f0 2022-03-07T23:31:58.922 INFO:teuthology.orchestra.run.smithi078.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-07T23:31:58.922 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:23 brd ff:ff:ff:ff:ff:ff 2022-03-07T23:31:58.923 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f1 2022-03-07T23:31:58.923 INFO:teuthology.orchestra.run.smithi078.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-03-07T23:31:58.923 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dc brd ff:ff:ff:ff:ff:ff 2022-03-07T23:31:58.923 INFO:teuthology.orchestra.run.smithi078.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2022-03-07T23:31:58.924 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dd brd ff:ff:ff:ff:ff:ff 2022-03-07T23:31:58.924 INFO:teuthology.orchestra.run.smithi078.stdout: inet 172.21.15.78/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2022-03-07T23:31:58.924 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft 42859sec preferred_lft 42859sec 2022-03-07T23:31:58.924 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::ec4:7aff:fe8f:d4dd/64 scope link 2022-03-07T23:31:58.924 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2022-03-07T23:31:58.925 INFO:teuthology.orchestra.run.smithi078.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-03-07T23:31:58.925 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 02:42:0a:54:51:5c brd ff:ff:ff:ff:ff:ff 2022-03-07T23:31:58.925 INFO:teuthology.orchestra.run.smithi078.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-03-07T23:31:58.925 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2022-03-07T23:31:58.927 DEBUG:teuthology.orchestra.run.smithi078:> which systemctl 2022-03-07T23:31:58.931 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/systemctl 2022-03-07T23:31:58.932 DEBUG:teuthology.orchestra.run.smithi078:> test -f /run/.containerenv -o -f /.dockerenv 2022-03-07T23:31:58.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-03-07T23:31:58.978 DEBUG:teuthology.orchestra.run.smithi078:> true 2022-03-07T23:31:59.008 DEBUG:teuthology.orchestra.run.smithi078:> sudo lsb_release -is 2022-03-07T23:31:59.056 INFO:teuthology.orchestra.run.smithi078.stdout:Ubuntu 2022-03-07T23:31:59.062 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-03-07T23:31:59.062 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-03-07T23:31:59.095 DEBUG:teuthology.orchestra.remote:smithi078:rook/deploy/examples/operator.yaml is 20KB 2022-03-07T23:31:59.146 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2022-03-07T23:31:59.147 DEBUG:teuthology.orchestra.run.smithi078:> dd if=rook/deploy/examples/operator.yaml of=/dev/stdout 2022-03-07T23:31:59.154 INFO:tasks.rook:Patching operator to use image rook/ceph:master 2022-03-07T23:31:59.195 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2022-03-07T23:31:59.195 DEBUG:teuthology.orchestra.run.smithi078:> dd of=operator.yaml 2022-03-07T23:31:59.208 INFO:tasks.rook:Deploying operator 2022-03-07T23:31:59.217 DEBUG:teuthology.orchestra.remote:smithi078:rook/deploy/examples/operator.yaml is 20KB 2022-03-07T23:31:59.239 DEBUG:teuthology.orchestra.remote:smithi078:rook/deploy/examples/operator.yaml is 20KB 2022-03-07T23:31:59.249 DEBUG:teuthology.orchestra.run.smithi078:> kubectl create -f rook/deploy/examples/crds.yaml -f rook/deploy/examples/common.yaml -f operator.yaml 2022-03-07T23:31:59.499 INFO:teuthology.orchestra.run.smithi078.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-03-07T23:31:59.552 INFO:teuthology.orchestra.run.smithi078.stdout:customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created 2022-03-07T23:31:59.562 INFO:teuthology.orchestra.run.smithi078.stdout:customresourcedefinition.apiextensions.k8s.io/cephbuckettopics.ceph.rook.io created 2022-03-07T23:31:59.595 INFO:teuthology.orchestra.run.smithi078.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-03-07T23:31:59.760 INFO:teuthology.orchestra.run.smithi078.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-03-07T23:31:59.797 INFO:teuthology.orchestra.run.smithi078.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-03-07T23:31:59.931 INFO:teuthology.orchestra.run.smithi078.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-03-07T23:31:59.938 INFO:teuthology.orchestra.run.smithi078.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemsubvolumegroups.ceph.rook.io created 2022-03-07T23:31:59.968 INFO:teuthology.orchestra.run.smithi078.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-03-07T23:31:59.989 INFO:teuthology.orchestra.run.smithi078.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-03-07T23:32:00.114 INFO:teuthology.orchestra.run.smithi078.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-03-07T23:32:00.150 INFO:teuthology.orchestra.run.smithi078.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-03-07T23:32:00.173 INFO:teuthology.orchestra.run.smithi078.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-03-07T23:32:00.200 INFO:teuthology.orchestra.run.smithi078.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-03-07T23:32:00.300 INFO:teuthology.orchestra.run.smithi078.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-03-07T23:32:00.334 INFO:teuthology.orchestra.run.smithi078.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-03-07T23:32:00.383 INFO:teuthology.orchestra.run.smithi078.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-03-07T23:32:00.492 INFO:teuthology.orchestra.run.smithi078.stdout:namespace/rook-ceph created 2022-03-07T23:32:00.516 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-07T23:32:00.551 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-03-07T23:32:00.642 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrole.rbac.authorization.k8s.io/psp:rook created 2022-03-07T23:32:00.651 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-07T23:32:00.685 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-03-07T23:32:00.836 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-07T23:32:00.842 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-07T23:32:00.894 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-07T23:32:01.046 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-07T23:32:01.051 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-07T23:32:01.079 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-07T23:32:01.085 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-07T23:32:01.092 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-03-07T23:32:01.101 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-03-07T23:32:01.121 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-07T23:32:01.205 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-03-07T23:32:01.208 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-03-07T23:32:01.230 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-03-07T23:32:01.246 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-03-07T23:32:01.255 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-07T23:32:01.271 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-07T23:32:01.347 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created 2022-03-07T23:32:01.350 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created 2022-03-07T23:32:01.376 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created 2022-03-07T23:32:01.384 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created 2022-03-07T23:32:01.393 INFO:teuthology.orchestra.run.smithi078.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created 2022-03-07T23:32:01.415 INFO:teuthology.orchestra.run.smithi078.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-03-07T23:32:01.415 INFO:teuthology.orchestra.run.smithi078.stdout:podsecuritypolicy.policy/00-rook-privileged created 2022-03-07T23:32:01.426 INFO:teuthology.orchestra.run.smithi078.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-03-07T23:32:01.434 INFO:teuthology.orchestra.run.smithi078.stdout:role.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-03-07T23:32:01.490 INFO:teuthology.orchestra.run.smithi078.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-03-07T23:32:01.494 INFO:teuthology.orchestra.run.smithi078.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-07T23:32:01.518 INFO:teuthology.orchestra.run.smithi078.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-07T23:32:01.574 INFO:teuthology.orchestra.run.smithi078.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-07T23:32:01.658 INFO:teuthology.orchestra.run.smithi078.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-07T23:32:01.661 INFO:teuthology.orchestra.run.smithi078.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-07T23:32:01.693 INFO:teuthology.orchestra.run.smithi078.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-03-07T23:32:01.718 INFO:teuthology.orchestra.run.smithi078.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-role-cfg created 2022-03-07T23:32:01.735 INFO:teuthology.orchestra.run.smithi078.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-03-07T23:32:01.837 INFO:teuthology.orchestra.run.smithi078.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-03-07T23:32:01.842 INFO:teuthology.orchestra.run.smithi078.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-03-07T23:32:01.861 INFO:teuthology.orchestra.run.smithi078.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created 2022-03-07T23:32:01.879 INFO:teuthology.orchestra.run.smithi078.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created 2022-03-07T23:32:01.884 INFO:teuthology.orchestra.run.smithi078.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-03-07T23:32:01.918 INFO:teuthology.orchestra.run.smithi078.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created 2022-03-07T23:32:01.926 INFO:teuthology.orchestra.run.smithi078.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-03-07T23:32:02.018 INFO:teuthology.orchestra.run.smithi078.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-03-07T23:32:02.021 INFO:teuthology.orchestra.run.smithi078.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created 2022-03-07T23:32:02.046 INFO:teuthology.orchestra.run.smithi078.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-03-07T23:32:02.090 INFO:teuthology.orchestra.run.smithi078.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-03-07T23:32:02.328 INFO:teuthology.orchestra.run.smithi078.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-03-07T23:32:02.332 INFO:teuthology.orchestra.run.smithi078.stdout:serviceaccount/rook-ceph-mgr created 2022-03-07T23:32:02.410 INFO:teuthology.orchestra.run.smithi078.stdout:serviceaccount/rook-ceph-osd created 2022-03-07T23:32:02.758 INFO:teuthology.orchestra.run.smithi078.stdout:serviceaccount/rook-ceph-purge-osd created 2022-03-07T23:32:02.763 INFO:teuthology.orchestra.run.smithi078.stdout:serviceaccount/rook-ceph-system created 2022-03-07T23:32:02.835 INFO:teuthology.orchestra.run.smithi078.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-03-07T23:32:03.367 INFO:teuthology.orchestra.run.smithi078.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-03-07T23:32:03.375 INFO:teuthology.orchestra.run.smithi078.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-03-07T23:32:03.567 INFO:teuthology.orchestra.run.smithi078.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-03-07T23:32:03.571 INFO:teuthology.orchestra.run.smithi078.stdout:configmap/rook-ceph-operator-config created 2022-03-07T23:32:03.828 INFO:teuthology.orchestra.run.smithi078.stdout:deployment.apps/rook-ceph-operator created 2022-03-07T23:32:03.834 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-07T23:32:04.004 INFO:teuthology.orchestra.run.smithi078.stderr:No resources found in rook-ceph namespace. 2022-03-07T23:32:14.008 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-07T23:32:14.071 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:32:14.072 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-operator-75dd789779-cft4c 0/1 ContainerCreating 0 10s 2022-03-07T23:32:24.075 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-07T23:32:24.139 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:32:24.140 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-operator-75dd789779-cft4c 0/1 ContainerCreating 0 20s 2022-03-07T23:32:34.143 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-07T23:32:34.209 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:32:34.209 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-operator-75dd789779-cft4c 0/1 ContainerCreating 0 30s 2022-03-07T23:32:44.212 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-03-07T23:32:44.280 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:32:44.281 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-operator-75dd789779-cft4c 1/1 Running 0 40s 2022-03-07T23:32:44.283 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph logs -f rook-ceph-operator-75dd789779-cft4c 2022-03-07T23:32:44.316 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2022-03-07T23:32:44.317 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/scratch_devs of=/dev/stdout 2022-03-07T23:32:44.325 DEBUG:teuthology.orchestra.run.smithi166:> set -ex 2022-03-07T23:32:44.326 DEBUG:teuthology.orchestra.run.smithi166:> dd if=/scratch_devs of=/dev/stdout 2022-03-07T23:32:44.335 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2022-03-07T23:32:44.336 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/scratch_devs of=/dev/stdout 2022-03-07T23:32:44.346 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-03-07T23:32:44.346 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-03-07T23:32:44.347 INFO:tasks.rook: override: [mon] debug mon = 20 2022-03-07T23:32:44.348 INFO:tasks.rook: override: [mon] debug ms = 1 2022-03-07T23:32:44.348 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-03-07T23:32:44.348 INFO:tasks.rook: override: [osd] debug ms = 1 2022-03-07T23:32:44.349 INFO:tasks.rook: override: [osd] debug osd = 20 2022-03-07T23:32:44.349 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-03-07T23:32:44.352 DEBUG:teuthology.orchestra.run.smithi078:> kubectl create -f - 2022-03-07T23:32:44.379 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:40.994462 I | rookcmd: starting Rook v1.8.0-alpha.0.381.g235781e28 with arguments '/usr/local/bin/rook ceph operator' 2022-03-07T23:32:44.380 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:40.994580 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO, --operator-image=, --service-account= 2022-03-07T23:32:44.380 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:40.994583 I | cephcmd: starting Rook-Ceph operator 2022-03-07T23:32:44.380 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.166479 I | cephcmd: base ceph version inside the rook operator image is "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)" 2022-03-07T23:32:44.381 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.173993 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-07T23:32:44.381 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.174020 I | operator: watching all namespaces for Ceph CRs 2022-03-07T23:32:44.381 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.174087 I | operator: setting up schemes 2022-03-07T23:32:44.381 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.177862 I | operator: setting up the controller-runtime manager 2022-03-07T23:32:44.381 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.382482 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-07T23:32:44.382 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.385711 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-07T23:32:44.382 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.385765 I | ceph-cluster-controller: successfully started 2022-03-07T23:32:44.382 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.385826 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-07T23:32:44.382 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.385857 I | ceph-crashcollector-controller: successfully started 2022-03-07T23:32:44.383 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.385888 I | ceph-block-pool-controller: successfully started 2022-03-07T23:32:44.383 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.385924 I | ceph-object-store-user-controller: successfully started 2022-03-07T23:32:44.383 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.385958 I | ceph-object-realm-controller: successfully started 2022-03-07T23:32:44.383 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.385972 I | ceph-object-zonegroup-controller: successfully started 2022-03-07T23:32:44.383 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.385984 I | ceph-object-zone-controller: successfully started 2022-03-07T23:32:44.384 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.386117 I | ceph-object-controller: successfully started 2022-03-07T23:32:44.384 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.386169 I | ceph-file-controller: successfully started 2022-03-07T23:32:44.384 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.386203 I | ceph-nfs-controller: successfully started 2022-03-07T23:32:44.384 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.386239 I | ceph-rbd-mirror-controller: successfully started 2022-03-07T23:32:44.384 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.386264 I | ceph-client-controller: successfully started 2022-03-07T23:32:44.385 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.386284 I | ceph-filesystem-mirror-controller: successfully started 2022-03-07T23:32:44.385 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.386311 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-07T23:32:44.385 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.386330 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-07T23:32:44.385 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.386349 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-07T23:32:44.386 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.386365 I | ceph-bucket-topic: successfully started 2022-03-07T23:32:44.386 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.386377 I | ceph-bucket-notification: successfully started 2022-03-07T23:32:44.386 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.386396 I | ceph-bucket-notification: successfully started 2022-03-07T23:32:44.386 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.386416 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-07T23:32:44.387 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.387769 I | operator: starting the controller-runtime manager 2022-03-07T23:32:44.387 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.492244 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-07T23:32:44.388 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.492267 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-07T23:32:44.388 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.492277 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-07T23:32:44.389 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.495968 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-07T23:32:44.389 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.498799 I | ceph-csi: CSI Ceph RBD driver disabled 2022-03-07T23:32:44.390 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.498821 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists 2022-03-07T23:32:44.390 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.501224 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists 2022-03-07T23:32:44.390 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.508250 I | ceph-csi: successfully removed CSI Ceph RBD driver 2022-03-07T23:32:44.391 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.508275 I | ceph-csi: CSI CephFS driver disabled 2022-03-07T23:32:44.391 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.508286 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists 2022-03-07T23:32:44.392 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.510606 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists 2022-03-07T23:32:44.392 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:41.516441 I | ceph-csi: successfully removed CSI CephFS driver 2022-03-07T23:32:45.235 INFO:teuthology.orchestra.run.smithi078.stdout:configmap/rook-config-override created 2022-03-07T23:32:45.239 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4 dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: true count: 3 skipUpgradeChecks: true 2022-03-07T23:32:45.239 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2022-03-07T23:32:45.240 DEBUG:teuthology.orchestra.run.smithi078:> dd of=cluster.yaml 2022-03-07T23:32:45.290 DEBUG:teuthology.orchestra.run.smithi078:> kubectl create -f cluster.yaml 2022-03-07T23:32:47.441 INFO:teuthology.orchestra.run.smithi078.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-03-07T23:32:47.448 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:47.449402 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-03-07T23:32:47.455 DEBUG:teuthology.orchestra.remote:smithi078:rook/deploy/examples/operator.yaml is 20KB 2022-03-07T23:32:47.465 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:47.467752 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-07T23:32:47.467 DEBUG:teuthology.orchestra.run.smithi078:> kubectl create -f rook/deploy/examples/toolbox.yaml 2022-03-07T23:32:47.506 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:47.508255 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster. 2022-03-07T23:32:47.523 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:47.525316 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-07T23:32:47.533 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:47.535499 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-07T23:32:47.601 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:47.602911 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-07T23:32:47.602 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:47.602937 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-07T23:32:47.602 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:47.602946 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-07T23:32:47.602 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:47.602954 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-07T23:32:47.603 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:47.602962 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-07T23:32:47.603 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:47.602971 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-07T23:32:47.604 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:47.602980 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-07T23:32:47.604 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:47.602989 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-07T23:32:47.604 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:47.602997 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-07T23:32:47.605 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:47.603006 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-07T23:32:47.605 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:47.603014 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-07T23:32:47.606 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:47.603023 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-07T23:32:47.606 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:47.603032 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-07T23:32:47.607 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:47.603038 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-07T23:32:47.607 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:47.603048 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-07T23:32:47.608 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:47.603057 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-07T23:32:47.608 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:47.603123 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-07T23:32:47.608 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:32:47.603144 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-07T23:32:47.793 INFO:teuthology.orchestra.run.smithi078.stdout:deployment.apps/rook-ceph-tools created 2022-03-07T23:32:47.799 INFO:tasks.rook:Waiting for tools container to start 2022-03-07T23:32:47.800 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:32:47.880 INFO:teuthology.orchestra.run.smithi078.stderr:No resources found in rook-ceph namespace. 2022-03-07T23:32:52.883 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:32:52.946 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:32:52.946 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 0/1 ContainerCreating 0 5s 2022-03-07T23:32:57.948 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:32:58.011 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:32:58.011 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 0/1 ContainerCreating 0 11s 2022-03-07T23:33:03.013 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:33:03.083 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:33:03.084 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 0/1 ContainerCreating 0 16s 2022-03-07T23:33:08.086 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:33:08.147 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:33:08.148 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 0/1 ContainerCreating 0 21s 2022-03-07T23:33:13.150 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:33:13.208 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:33:13.209 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 0/1 ContainerCreating 0 26s 2022-03-07T23:33:18.211 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:33:18.277 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:33:18.278 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 0/1 ContainerCreating 0 31s 2022-03-07T23:33:23.280 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:33:23.339 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:33:23.340 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 0/1 ContainerCreating 0 36s 2022-03-07T23:33:28.342 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:33:28.412 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:33:28.413 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 0/1 ContainerCreating 0 41s 2022-03-07T23:33:33.415 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:33:33.484 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:33:33.484 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 0/1 ContainerCreating 0 46s 2022-03-07T23:33:38.487 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:33:38.548 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:33:38.548 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 0/1 ContainerCreating 0 51s 2022-03-07T23:33:43.551 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:33:43.610 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:33:43.610 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 0/1 ContainerCreating 0 56s 2022-03-07T23:33:48.612 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:33:48.689 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:33:48.690 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 0/1 ContainerCreating 0 61s 2022-03-07T23:33:53.692 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:33:53.758 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:33:53.759 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 0/1 ContainerCreating 0 66s 2022-03-07T23:33:57.741 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:33:57.740435 I | ceph-spec: detected ceph image version: "17.0.0-11002 quincy" 2022-03-07T23:33:57.741 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:33:57.740464 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-07T23:33:57.741 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:33:57.740485 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11002 quincy", pursuing 2022-03-07T23:33:57.805 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:33:57.805587 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11002 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4" 2022-03-07T23:33:57.906 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:33:57.906220 I | op-mon: start running mons 2022-03-07T23:33:58.024 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:33:58.023797 I | op-mon: creating mon secrets for a new cluster 2022-03-07T23:33:58.046 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:33:58.046103 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-03-07T23:33:58.050 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:33:58.048748 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1] 2022-03-07T23:33:58.761 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:33:58.767 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:33:58.767808 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T23:33:58.768 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:33:58.768042 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T23:33:58.825 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:33:58.825 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 0/1 ContainerCreating 0 71s 2022-03-07T23:34:00.367 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:00.367917 I | op-mon: targeting the mon count 3 2022-03-07T23:34:00.387 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:00.387292 I | op-mon: created canary deployment rook-ceph-mon-a-canary 2022-03-07T23:34:00.409 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:00.409404 I | op-mon: created canary deployment rook-ceph-mon-b-canary 2022-03-07T23:34:00.441 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:00.437642 I | op-mon: created canary deployment rook-ceph-mon-c-canary 2022-03-07T23:34:01.168 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:01.168741 I | op-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi170 2022-03-07T23:34:01.169 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:01.168774 I | op-mon: mon a assigned to node smithi170 2022-03-07T23:34:01.366 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:01.365824 I | op-mon: canary monitor deployment rook-ceph-mon-b-canary scheduled to smithi166 2022-03-07T23:34:01.366 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:01.365857 I | op-mon: mon b assigned to node smithi166 2022-03-07T23:34:01.566 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:01.565952 I | op-mon: canary monitor deployment rook-ceph-mon-c-canary scheduled to smithi078 2022-03-07T23:34:01.566 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:01.565981 I | op-mon: mon c assigned to node smithi078 2022-03-07T23:34:01.570 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:01.570870 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-03-07T23:34:01.576 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:01.577100 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-b-canary" 2022-03-07T23:34:01.689 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:01.689391 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-c-canary" 2022-03-07T23:34:01.858 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:01.857168 I | op-mon: creating mon a 2022-03-07T23:34:02.043 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:02.041974 I | op-mon: mon "a" endpoint is [v2:10.105.146.196:3300,v1:10.105.146.196:6789] 2022-03-07T23:34:02.392 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:02.391230 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-07T23:34:02.392 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:02.391396 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-03-07T23:34:02.392 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:02.391547 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.105.146.196:6789"]}] data:a=10.105.146.196:6789 mapping:{"node":{"a":{"Name":"smithi170","Hostname":"smithi170","Address":"172.21.15.170"},"b":{"Name":"smithi166","Hostname":"smithi166","Address":"172.21.15.166"},"c":{"Name":"smithi078","Hostname":"smithi078","Address":"172.21.15.78"}}} maxMonId:-1] 2022-03-07T23:34:02.967 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:02.967197 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T23:34:02.968 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:02.967520 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T23:34:03.398 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:03.398893 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-03-07T23:34:03.568 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:03.567983 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a" 2022-03-07T23:34:03.828 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:34:03.889 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:34:03.890 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 0/1 ContainerCreating 0 76s 2022-03-07T23:34:04.366 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:04.366435 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.105.146.196:6789"]}] data:a=10.105.146.196:6789 mapping:{"node":{"a":{"Name":"smithi170","Hostname":"smithi170","Address":"172.21.15.170"},"b":{"Name":"smithi166","Hostname":"smithi166","Address":"172.21.15.166"},"c":{"Name":"smithi078","Hostname":"smithi078","Address":"172.21.15.78"}}} maxMonId:0] 2022-03-07T23:34:04.367 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:04.366470 I | op-mon: waiting for mon quorum with [a] 2022-03-07T23:34:04.568 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:04.568918 I | op-mon: mon a is not yet running 2022-03-07T23:34:04.569 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:04.568950 I | op-mon: mons running: [] 2022-03-07T23:34:07.601 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:07.601363 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789 2022-03-07T23:34:07.601 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:07.601427 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-07T23:34:07.602 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:07.601437 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-07T23:34:07.606 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:07.603156 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-07T23:34:07.606 INFO:tasks.rook.operator.smithi078.stdout:I0307 23:34:07.603229 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-07T23:34:08.892 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:34:08.957 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:34:08.958 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 0/1 ContainerCreating 0 81s 2022-03-07T23:34:13.960 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:34:14.029 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:34:14.029 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 0/1 ContainerCreating 0 87s 2022-03-07T23:34:19.032 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:34:19.093 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:34:19.094 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 0/1 ContainerCreating 0 92s 2022-03-07T23:34:24.097 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:34:24.796 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:24.796916 I | op-mon: mons running: [a] 2022-03-07T23:34:24.797 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:34:24.797 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 0/1 ContainerCreating 0 97s 2022-03-07T23:34:26.044 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:26.044298 I | op-mon: Monitors in quorum: [a] 2022-03-07T23:34:26.045 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:26.044329 I | op-mon: mons created: 1 2022-03-07T23:34:26.275 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:26.276196 I | op-mon: waiting for mon quorum with [a] 2022-03-07T23:34:26.355 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:26.356285 I | op-mon: mons running: [a] 2022-03-07T23:34:26.582 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:26.582252 I | op-mon: Monitors in quorum: [a] 2022-03-07T23:34:26.582 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:26.582281 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T23:34:26.875 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:26.874578 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T23:34:26.875 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:26.874603 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T23:34:27.169 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:27.169499 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T23:34:27.169 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:27.169519 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T23:34:27.469 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:27.469049 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T23:34:27.469 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:27.469084 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-07T23:34:27.767 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:27.767984 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-07T23:34:27.768 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:27.768026 I | op-config: deleting "log file" option from the mon configuration database 2022-03-07T23:34:27.982 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:27.982436 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-07T23:34:27.982 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:27.982467 I | op-mon: creating mon b 2022-03-07T23:34:29.799 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:34:29.834 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:29.834516 I | op-mon: mon "a" endpoint is [v2:10.105.146.196:3300,v1:10.105.146.196:6789] 2022-03-07T23:34:30.145 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:34:30.146 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 0/1 ContainerCreating 0 103s 2022-03-07T23:34:30.538 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:30.539038 I | op-mon: mon "b" endpoint is [v2:10.98.222.16:3300,v1:10.98.222.16:6789] 2022-03-07T23:34:31.310 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:31.310409 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.105.146.196:6789","10.98.222.16:6789"]}] data:a=10.105.146.196:6789,b=10.98.222.16:6789 mapping:{"node":{"a":{"Name":"smithi170","Hostname":"smithi170","Address":"172.21.15.170"},"b":{"Name":"smithi166","Hostname":"smithi166","Address":"172.21.15.166"},"c":{"Name":"smithi078","Hostname":"smithi078","Address":"172.21.15.78"}}} maxMonId:0] 2022-03-07T23:34:31.768 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:31.769075 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T23:34:31.769 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:31.769373 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T23:34:31.849 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:31.849769 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-07T23:34:31.862 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:31.862533 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-07T23:34:31.862 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:31.862566 I | op-mon: waiting for mon quorum with [a b] 2022-03-07T23:34:32.055 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:32.056330 I | op-mon: mons running: [a b] 2022-03-07T23:34:33.077 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:33.077775 I | op-mon: Monitors in quorum: [a] 2022-03-07T23:34:33.268 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:33.268464 I | op-mon: updating maxMonID from 0 to 1 after committing mon "b" 2022-03-07T23:34:33.620 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:33.620712 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.105.146.196:6789","10.98.222.16:6789"]}] data:a=10.105.146.196:6789,b=10.98.222.16:6789 mapping:{"node":{"a":{"Name":"smithi170","Hostname":"smithi170","Address":"172.21.15.170"},"b":{"Name":"smithi166","Hostname":"smithi166","Address":"172.21.15.166"},"c":{"Name":"smithi078","Hostname":"smithi078","Address":"172.21.15.78"}}} maxMonId:1] 2022-03-07T23:34:33.621 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:33.620747 I | op-mon: waiting for mon quorum with [a b] 2022-03-07T23:34:33.631 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:33.632380 I | op-mon: mons running: [a b] 2022-03-07T23:34:34.333 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:34.333363 I | op-mon: Monitors in quorum: [a] 2022-03-07T23:34:34.333 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:34.333393 I | op-mon: mons created: 2 2022-03-07T23:34:34.554 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:34.554577 I | op-mon: waiting for mon quorum with [a b] 2022-03-07T23:34:34.569 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:34.570627 I | op-mon: mons running: [a b] 2022-03-07T23:34:35.071 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:35.072164 W | op-mon: failed to find initial monitor b in mon map 2022-03-07T23:34:35.072 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:35.072199 W | op-mon: monitor b is not in quorum list 2022-03-07T23:34:35.148 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:34:35.218 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:34:35.219 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 0/1 ContainerCreating 0 108s 2022-03-07T23:34:40.090 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:40.090897 I | op-mon: mon b is not yet running 2022-03-07T23:34:40.090 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:40.090930 I | op-mon: mons running: [a] 2022-03-07T23:34:40.222 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:34:40.288 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:34:40.288 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 0/1 ContainerCreating 0 113s 2022-03-07T23:34:41.439 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.438043 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-07T23:34:41.439 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.438068 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-07T23:34:41.439 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.438074 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-07T23:34:41.440 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.438077 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-07T23:34:41.440 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.438082 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-07T23:34:41.440 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.438085 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-07T23:34:41.440 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.438089 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-07T23:34:41.441 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.438092 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-07T23:34:41.441 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.438095 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-07T23:34:41.441 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.438098 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-07T23:34:41.441 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.438104 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-07T23:34:41.442 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.438109 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-07T23:34:41.442 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.438115 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-07T23:34:41.442 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.438120 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-07T23:34:41.442 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.438125 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-07T23:34:41.443 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.438131 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-07T23:34:41.443 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.438136 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-07T23:34:41.443 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.438141 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-07T23:34:41.443 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.438147 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-07T23:34:41.443 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.438153 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-07T23:34:41.444 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.438158 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-07T23:34:41.444 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.438165 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-07T23:34:41.444 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.438170 I | ceph-csi: Kubernetes version is 1.23 2022-03-07T23:34:41.444 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.438177 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-07T23:34:41.445 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.438181 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-07T23:34:41.446 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.444968 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-07T23:34:41.453 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.453314 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-07T23:34:41.454 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.453339 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-07T23:34:41.454 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.453346 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-07T23:34:41.454 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.453352 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-07T23:34:41.456 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.453358 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-07T23:34:41.456 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.453364 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-07T23:34:41.456 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.453371 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-07T23:34:41.540 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.541285 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-07T23:34:41.541 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.541308 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-07T23:34:41.541 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.541314 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-07T23:34:41.607 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.607781 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-07T23:34:41.736 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.732284 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-07T23:34:41.736 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.732308 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-07T23:34:41.736 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.732315 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-07T23:34:41.874 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.874640 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-07T23:34:41.874 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.874666 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-07T23:34:41.875 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.874684 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-07T23:34:41.925 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:41.925213 I | ceph-csi: successfully started CSI CephFS driver 2022-03-07T23:34:42.542 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:42.541764 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-07T23:34:42.789 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:42.789765 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-07T23:34:42.789 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:42.789803 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-07T23:34:43.117 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:43.117794 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-07T23:34:45.115 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:45.116192 I | op-mon: mon b is not yet running 2022-03-07T23:34:45.117 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:45.116220 I | op-mon: mons running: [a] 2022-03-07T23:34:45.292 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:34:45.360 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:34:45.361 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 0/1 ContainerCreating 0 118s 2022-03-07T23:34:50.133 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:50.134135 I | op-mon: mon b is not yet running 2022-03-07T23:34:50.133 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:50.134152 I | op-mon: mons running: [a] 2022-03-07T23:34:50.363 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:34:50.428 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:34:50.548 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 0/1 ContainerCreating 0 2m3s 2022-03-07T23:34:55.167 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:55.168128 I | op-mon: mon b is not yet running 2022-03-07T23:34:55.167 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:34:55.168166 I | op-mon: mons running: [a] 2022-03-07T23:34:55.551 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:34:55.617 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:34:55.617 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 0/1 ContainerCreating 0 2m8s 2022-03-07T23:35:00.191 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:00.190932 I | op-mon: mon b is not yet running 2022-03-07T23:35:00.192 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:00.190964 I | op-mon: mons running: [a] 2022-03-07T23:35:00.620 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:35:00.681 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:35:00.682 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 0/1 ContainerCreating 0 2m13s 2022-03-07T23:35:05.534 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:05.535363 I | op-mon: mon b is not yet running 2022-03-07T23:35:05.534 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:05.535431 I | op-mon: mons running: [a] 2022-03-07T23:35:05.684 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:35:06.056 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:35:06.057 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 0/1 ContainerCreating 0 2m19s 2022-03-07T23:35:10.700 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:10.701633 I | op-mon: mons running: [a b] 2022-03-07T23:35:11.059 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-03-07T23:35:11.139 INFO:teuthology.orchestra.run.smithi078.stdout:NAME READY STATUS RESTARTS AGE 2022-03-07T23:35:11.140 INFO:teuthology.orchestra.run.smithi078.stdout:rook-ceph-tools-d6d7c985c-jnk6p 1/1 Running 0 2m24s 2022-03-07T23:35:11.141 INFO:tasks.rook:Waiting for mgr/rook orchestrator to be available 2022-03-07T23:35:11.141 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch status -f json 2022-03-07T23:35:16.288 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:35:16.289 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2022-03-07T23:35:16.289 INFO:teuthology.orchestra.run.smithi078.stderr:pg stat 2022-03-07T23:35:16.290 INFO:teuthology.orchestra.run.smithi078.stderr:pg getmap 2022-03-07T23:35:16.290 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-07T23:35:16.291 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-07T23:35:16.291 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump_pools_json 2022-03-07T23:35:16.292 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls-by-pool [...] 2022-03-07T23:35:16.292 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls-by-primary [] [...] 2022-03-07T23:35:16.293 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls-by-osd [] [...] 2022-03-07T23:35:16.293 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls [] [...] 2022-03-07T23:35:16.294 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-07T23:35:16.294 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2022-03-07T23:35:16.296 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-07T23:35:16.297 INFO:teuthology.orchestra.run.smithi078.stderr:command terminated with exit code 22 2022-03-07T23:35:16.983 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:16.984332 I | op-mon: Monitors in quorum: [a b] 2022-03-07T23:35:16.983 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:16.984357 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T23:35:17.226 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:17.227791 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T23:35:17.226 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:17.227830 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T23:35:17.477 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:17.479228 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T23:35:17.478 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:17.479248 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T23:35:17.702 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:17.703822 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T23:35:17.703 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:17.703857 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-07T23:35:17.990 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:17.991281 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-07T23:35:17.990 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:17.991309 I | op-config: deleting "log file" option from the mon configuration database 2022-03-07T23:35:18.231 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:18.232069 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-07T23:35:18.231 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:18.232100 I | op-mon: creating mon c 2022-03-07T23:35:18.366 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:18.367924 I | op-mon: mon "a" endpoint is [v2:10.105.146.196:3300,v1:10.105.146.196:6789] 2022-03-07T23:35:18.511 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:18.512699 I | op-mon: mon "b" endpoint is [v2:10.98.222.16:3300,v1:10.98.222.16:6789] 2022-03-07T23:35:18.545 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:18.546692 I | op-mon: mon "c" endpoint is [v2:10.96.111.213:3300,v1:10.96.111.213:6789] 2022-03-07T23:35:18.663 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:18.664673 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.105.146.196:6789","10.98.222.16:6789","10.96.111.213:6789"]}] data:b=10.98.222.16:6789,c=10.96.111.213:6789,a=10.105.146.196:6789 mapping:{"node":{"a":{"Name":"smithi170","Hostname":"smithi170","Address":"172.21.15.170"},"b":{"Name":"smithi166","Hostname":"smithi166","Address":"172.21.15.166"},"c":{"Name":"smithi078","Hostname":"smithi078","Address":"172.21.15.78"}}} maxMonId:1] 2022-03-07T23:35:19.236 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:19.237524 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T23:35:19.240 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:19.237792 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T23:35:19.649 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:19.650775 I | op-mon: 2 of 3 expected mon deployments exist. creating new deployment(s). 2022-03-07T23:35:19.655 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:19.656749 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-07T23:35:19.667 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:19.669542 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-07T23:35:19.671 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:19.673793 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-07T23:35:19.680 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:19.681992 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-07T23:35:19.863 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:19.865018 I | op-mon: updating maxMonID from 1 to 2 after committing mon "c" 2022-03-07T23:35:20.837 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:20.839314 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.105.146.196:6789","10.98.222.16:6789","10.96.111.213:6789"]}] data:a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 mapping:{"node":{"a":{"Name":"smithi170","Hostname":"smithi170","Address":"172.21.15.170"},"b":{"Name":"smithi166","Hostname":"smithi166","Address":"172.21.15.166"},"c":{"Name":"smithi078","Hostname":"smithi078","Address":"172.21.15.78"}}} maxMonId:2] 2022-03-07T23:35:20.838 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:20.839342 I | op-mon: waiting for mon quorum with [a b c] 2022-03-07T23:35:21.350 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:21.352352 I | op-mon: mon c is not yet running 2022-03-07T23:35:21.351 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:21.352389 I | op-mon: mons running: [a b] 2022-03-07T23:35:21.642 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:21.643902 I | op-mon: Monitors in quorum: [a b] 2022-03-07T23:35:21.642 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:21.643925 I | op-mon: mons created: 3 2022-03-07T23:35:21.916 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:21.918262 I | op-mon: waiting for mon quorum with [a b c] 2022-03-07T23:35:21.934 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:21.935701 I | op-mon: mon c is not yet running 2022-03-07T23:35:21.934 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:21.935729 I | op-mon: mons running: [a b] 2022-03-07T23:35:26.299 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch status -f json 2022-03-07T23:35:26.770 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:35:26.771 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2022-03-07T23:35:26.771 INFO:teuthology.orchestra.run.smithi078.stderr:pg stat 2022-03-07T23:35:26.771 INFO:teuthology.orchestra.run.smithi078.stderr:pg getmap 2022-03-07T23:35:26.772 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-07T23:35:26.772 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-07T23:35:26.772 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump_pools_json 2022-03-07T23:35:26.772 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls-by-pool [...] 2022-03-07T23:35:26.773 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls-by-primary [] [...] 2022-03-07T23:35:26.773 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls-by-osd [] [...] 2022-03-07T23:35:26.773 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls [] [...] 2022-03-07T23:35:26.773 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-07T23:35:26.773 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2022-03-07T23:35:26.775 INFO:teuthology.orchestra.run.smithi078.stderr:command terminated with exit code 22 2022-03-07T23:35:26.777 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-07T23:35:26.963 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:26.960857 I | op-mon: mon c is not yet running 2022-03-07T23:35:26.963 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:26.960906 I | op-mon: mons running: [a b] 2022-03-07T23:35:31.978 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:31.980256 I | op-mon: mon c is not yet running 2022-03-07T23:35:31.978 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:31.980284 I | op-mon: mons running: [a b] 2022-03-07T23:35:36.779 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch status -f json 2022-03-07T23:35:36.998 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:37.000543 I | op-mon: mon c is not yet running 2022-03-07T23:35:36.999 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:37.000574 I | op-mon: mons running: [a b] 2022-03-07T23:35:37.259 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:35:37.260 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2022-03-07T23:35:37.260 INFO:teuthology.orchestra.run.smithi078.stderr:pg stat 2022-03-07T23:35:37.260 INFO:teuthology.orchestra.run.smithi078.stderr:pg getmap 2022-03-07T23:35:37.261 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-07T23:35:37.261 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-07T23:35:37.262 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump_pools_json 2022-03-07T23:35:37.262 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls-by-pool [...] 2022-03-07T23:35:37.262 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls-by-primary [] [...] 2022-03-07T23:35:37.263 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls-by-osd [] [...] 2022-03-07T23:35:37.263 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls [] [...] 2022-03-07T23:35:37.264 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-07T23:35:37.264 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2022-03-07T23:35:37.265 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-07T23:35:37.266 INFO:teuthology.orchestra.run.smithi078.stderr:command terminated with exit code 22 2022-03-07T23:35:42.163 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:42.165484 I | op-mon: mon c is not yet running 2022-03-07T23:35:42.164 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:42.165509 I | op-mon: mons running: [a b] 2022-03-07T23:35:47.268 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch status -f json 2022-03-07T23:35:47.445 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:47.447667 I | op-mon: mon c is not yet running 2022-03-07T23:35:47.446 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:47.447691 I | op-mon: mons running: [a b] 2022-03-07T23:35:48.116 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:35:48.117 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2022-03-07T23:35:48.117 INFO:teuthology.orchestra.run.smithi078.stderr:pg stat 2022-03-07T23:35:48.117 INFO:teuthology.orchestra.run.smithi078.stderr:pg getmap 2022-03-07T23:35:48.117 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-07T23:35:48.117 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-07T23:35:48.118 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump_pools_json 2022-03-07T23:35:48.118 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls-by-pool [...] 2022-03-07T23:35:48.118 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls-by-primary [] [...] 2022-03-07T23:35:48.118 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls-by-osd [] [...] 2022-03-07T23:35:48.119 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls [] [...] 2022-03-07T23:35:48.119 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-07T23:35:48.119 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2022-03-07T23:35:48.121 INFO:teuthology.orchestra.run.smithi078.stderr:command terminated with exit code 22 2022-03-07T23:35:48.123 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-07T23:35:53.737 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:53.739597 I | op-mon: mon c is not yet running 2022-03-07T23:35:53.737 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:53.739618 I | op-mon: mons running: [a b] 2022-03-07T23:35:58.125 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch status -f json 2022-03-07T23:35:59.880 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:59.882602 I | op-mon: mon c is not yet running 2022-03-07T23:35:59.880 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:35:59.882627 I | op-mon: mons running: [a b] 2022-03-07T23:36:00.264 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:36:00.265 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2022-03-07T23:36:00.265 INFO:teuthology.orchestra.run.smithi078.stderr:pg stat 2022-03-07T23:36:00.265 INFO:teuthology.orchestra.run.smithi078.stderr:pg getmap 2022-03-07T23:36:00.265 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-07T23:36:00.266 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-07T23:36:00.266 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump_pools_json 2022-03-07T23:36:00.266 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls-by-pool [...] 2022-03-07T23:36:00.266 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls-by-primary [] [...] 2022-03-07T23:36:00.266 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls-by-osd [] [...] 2022-03-07T23:36:00.267 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls [] [...] 2022-03-07T23:36:00.267 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-07T23:36:00.267 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2022-03-07T23:36:00.268 INFO:teuthology.orchestra.run.smithi078.stderr:command terminated with exit code 22 2022-03-07T23:36:00.268 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-07T23:36:04.896 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:04.898165 I | op-mon: mon c is not yet running 2022-03-07T23:36:04.896 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:04.898187 I | op-mon: mons running: [a b] 2022-03-07T23:36:10.270 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch status -f json 2022-03-07T23:36:11.513 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:36:11.514 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2022-03-07T23:36:11.514 INFO:teuthology.orchestra.run.smithi078.stderr:pg stat 2022-03-07T23:36:11.515 INFO:teuthology.orchestra.run.smithi078.stderr:pg getmap 2022-03-07T23:36:11.515 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-07T23:36:11.515 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-07T23:36:11.515 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump_pools_json 2022-03-07T23:36:11.516 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls-by-pool [...] 2022-03-07T23:36:11.516 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls-by-primary [] [...] 2022-03-07T23:36:11.516 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls-by-osd [] [...] 2022-03-07T23:36:11.516 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls [] [...] 2022-03-07T23:36:11.517 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-07T23:36:11.517 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2022-03-07T23:36:11.518 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-07T23:36:11.518 INFO:teuthology.orchestra.run.smithi078.stderr:command terminated with exit code 22 2022-03-07T23:36:11.819 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:11.822040 I | op-mon: mon c is not yet running 2022-03-07T23:36:11.820 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:11.822069 I | op-mon: mons running: [a b] 2022-03-07T23:36:17.191 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:17.193656 I | op-mon: mon c is not yet running 2022-03-07T23:36:17.191 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:17.193680 I | op-mon: mons running: [a b] 2022-03-07T23:36:21.520 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch status -f json 2022-03-07T23:36:22.525 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:22.527739 I | op-mon: mons running: [a b c] 2022-03-07T23:36:25.971 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:25.974123 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T23:36:25.972 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:25.974155 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T23:36:26.125 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:36:26.126 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2022-03-07T23:36:26.126 INFO:teuthology.orchestra.run.smithi078.stderr:pg stat 2022-03-07T23:36:26.126 INFO:teuthology.orchestra.run.smithi078.stderr:pg getmap 2022-03-07T23:36:26.127 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-07T23:36:26.127 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-07T23:36:26.127 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump_pools_json 2022-03-07T23:36:26.127 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls-by-pool [...] 2022-03-07T23:36:26.128 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls-by-primary [] [...] 2022-03-07T23:36:26.128 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls-by-osd [] [...] 2022-03-07T23:36:26.128 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls [] [...] 2022-03-07T23:36:26.129 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-07T23:36:26.129 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2022-03-07T23:36:26.130 INFO:teuthology.orchestra.run.smithi078.stderr:command terminated with exit code 22 2022-03-07T23:36:26.133 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-07T23:36:26.237 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:26.240055 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T23:36:26.237 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:26.240083 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T23:36:26.467 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:26.469857 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T23:36:26.467 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:26.469876 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T23:36:26.686 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:26.689368 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T23:36:26.687 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:26.689399 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-07T23:36:26.903 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:26.906045 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-07T23:36:26.904 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:26.906072 I | op-config: deleting "log file" option from the mon configuration database 2022-03-07T23:36:27.774 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:27.777397 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-07T23:36:27.775 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:27.777451 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-07T23:36:28.166 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:28.169225 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-07T23:36:28.747 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:28.750474 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-07T23:36:29.397 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:29.399713 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-07T23:36:30.095 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:30.097730 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-07T23:36:30.095 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:30.097750 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-07T23:36:30.744 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:30.747702 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-07T23:36:30.985 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:30.987625 I | cephclient: successfully enabled msgr2 protocol 2022-03-07T23:36:30.985 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:30.987663 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-07T23:36:30.985 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:30.987671 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-07T23:36:31.316 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:31.319160 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-07T23:36:31.334 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:31.336854 I | op-mgr: start running mgr 2022-03-07T23:36:31.334 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:31.336891 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-07T23:36:32.023 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:32.026358 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi078": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T23:36:32.102 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:32.104655 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi078": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T23:36:34.213 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:34.216221 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi170": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T23:36:36.135 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch status -f json 2022-03-07T23:36:36.710 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:36:36.711 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2022-03-07T23:36:36.711 INFO:teuthology.orchestra.run.smithi078.stderr:pg stat 2022-03-07T23:36:36.712 INFO:teuthology.orchestra.run.smithi078.stderr:pg getmap 2022-03-07T23:36:36.712 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-07T23:36:36.712 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-07T23:36:36.712 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump_pools_json 2022-03-07T23:36:36.713 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls-by-pool [...] 2022-03-07T23:36:36.713 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls-by-primary [] [...] 2022-03-07T23:36:36.713 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls-by-osd [] [...] 2022-03-07T23:36:36.713 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls [] [...] 2022-03-07T23:36:36.713 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-07T23:36:36.714 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2022-03-07T23:36:36.715 INFO:teuthology.orchestra.run.smithi078.stderr:command terminated with exit code 22 2022-03-07T23:36:36.717 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-07T23:36:46.718 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch status -f json 2022-03-07T23:36:47.260 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:36:47.261 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2022-03-07T23:36:47.261 INFO:teuthology.orchestra.run.smithi078.stderr:pg stat 2022-03-07T23:36:47.261 INFO:teuthology.orchestra.run.smithi078.stderr:pg getmap 2022-03-07T23:36:47.261 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump [all|summary|sum|delta|pools|osds|pgs|pgs_brief...] 2022-03-07T23:36:47.262 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump_json [all|summary|sum|pools|osds|pgs...] 2022-03-07T23:36:47.262 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump_pools_json 2022-03-07T23:36:47.262 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls-by-pool [...] 2022-03-07T23:36:47.262 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls-by-primary [] [...] 2022-03-07T23:36:47.262 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls-by-osd [] [...] 2022-03-07T23:36:47.263 INFO:teuthology.orchestra.run.smithi078.stderr:pg ls [] [...] 2022-03-07T23:36:47.263 INFO:teuthology.orchestra.run.smithi078.stderr:pg dump_stuck [inactive|unclean|stale|undersized|degraded...] [] 2022-03-07T23:36:47.263 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2022-03-07T23:36:47.265 INFO:teuthology.orchestra.run.smithi078.stderr:command terminated with exit code 22 2022-03-07T23:36:47.266 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-03-07T23:36:52.912 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:52.915448 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-03-07T23:36:52.916 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:52.919861 I | op-mgr: setting services to point to mgr "a" 2022-03-07T23:36:52.996 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:52.999731 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-07T23:36:53.006 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:53.000709 I | op-mgr: successful modules: balancer 2022-03-07T23:36:53.009 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:53.013314 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-07T23:36:53.010 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:53.013355 W | op-osd: useAllNodes is set to false and no nodes, storageClassDevicesets or volumeSources are specified, no OSD pods are going to be created 2022-03-07T23:36:53.010 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:53.013366 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-07T23:36:53.031 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:53.035329 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-07T23:36:53.043 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:53.046156 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs 2022-03-07T23:36:53.043 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:53.046187 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-07T23:36:53.043 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:53.046193 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-07T23:36:53.556 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:53.559741 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-07T23:36:53.557 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:53.559762 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-07T23:36:53.566 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:53.569359 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph" 2022-03-07T23:36:53.567 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:53.569396 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph" 2022-03-07T23:36:53.567 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:53.569413 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph" 2022-03-07T23:36:53.727 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:53.731130 I | op-mgr: successful modules: prometheus 2022-03-07T23:36:54.835 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:54.838406 I | op-mgr: successful modules: dashboard 2022-03-07T23:36:54.838 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:36:54.839214 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-07T23:36:57.268 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch status -f json 2022-03-07T23:37:08.383 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:08.386503 I | op-mgr: successful modules: orchestrator modules 2022-03-07T23:37:08.475 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:37:08.475 INFO:teuthology.orchestra.run.smithi078.stdout:{"available": true, "backend": "rook"} 2022-03-07T23:37:08.506 INFO:tasks.rook: mgr/rook orchestrator is active 2022-03-07T23:37:08.506 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph config set mgr mgr/rook/storage_class scratch 2022-03-07T23:37:09.191 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch apply osd --all-available-devices 2022-03-07T23:37:09.927 INFO:teuthology.orchestra.run.smithi078.stdout:['Success'] 2022-03-07T23:37:09.928 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:09.931683 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-07T23:37:09.928 INFO:tasks.rook.operator.smithi078.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", AllowUnsupported: true}, 2022-03-07T23:37:09.928 INFO:tasks.rook.operator.smithi078.stdout: Storage: v1.StorageScopeSpec{ 2022-03-07T23:37:09.929 INFO:tasks.rook.operator.smithi078.stdout: ... // 3 identical fields 2022-03-07T23:37:09.930 INFO:tasks.rook.operator.smithi078.stdout: Config: nil, 2022-03-07T23:37:09.930 INFO:tasks.rook.operator.smithi078.stdout: Selection: {}, 2022-03-07T23:37:09.931 INFO:tasks.rook.operator.smithi078.stdout:- StorageClassDeviceSets: nil, 2022-03-07T23:37:09.931 INFO:tasks.rook.operator.smithi078.stdout:+ StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-07T23:37:09.931 INFO:tasks.rook.operator.smithi078.stdout:+ { 2022-03-07T23:37:09.931 INFO:tasks.rook.operator.smithi078.stdout:+ Name: "smithi078-lv-1", 2022-03-07T23:37:09.931 INFO:tasks.rook.operator.smithi078.stdout:+ Count: 1, 2022-03-07T23:37:09.932 INFO:tasks.rook.operator.smithi078.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-07T23:37:09.932 INFO:tasks.rook.operator.smithi078.stdout:+ }, 2022-03-07T23:37:09.932 INFO:tasks.rook.operator.smithi078.stdout:+ { 2022-03-07T23:37:09.932 INFO:tasks.rook.operator.smithi078.stdout:+ Name: "smithi078-lv-2", 2022-03-07T23:37:09.933 INFO:tasks.rook.operator.smithi078.stdout:+ Count: 1, 2022-03-07T23:37:09.933 INFO:tasks.rook.operator.smithi078.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-07T23:37:09.933 INFO:tasks.rook.operator.smithi078.stdout:+ }, 2022-03-07T23:37:09.933 INFO:tasks.rook.operator.smithi078.stdout:+ { 2022-03-07T23:37:09.933 INFO:tasks.rook.operator.smithi078.stdout:+ Name: "smithi078-lv-3", 2022-03-07T23:37:09.939 INFO:tasks.rook.operator.smithi078.stdout:+ Count: 1, 2022-03-07T23:37:09.939 INFO:tasks.rook.operator.smithi078.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-07T23:37:09.939 INFO:tasks.rook.operator.smithi078.stdout:+ }, 2022-03-07T23:37:09.940 INFO:tasks.rook.operator.smithi078.stdout:+ { 2022-03-07T23:37:09.940 INFO:tasks.rook.operator.smithi078.stdout:+ Name: "smithi078-lv-4", 2022-03-07T23:37:09.940 INFO:tasks.rook.operator.smithi078.stdout:+ Count: 1, 2022-03-07T23:37:09.941 INFO:tasks.rook.operator.smithi078.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-07T23:37:09.941 INFO:tasks.rook.operator.smithi078.stdout:+ }, 2022-03-07T23:37:09.941 INFO:tasks.rook.operator.smithi078.stdout:+ { 2022-03-07T23:37:09.941 INFO:tasks.rook.operator.smithi078.stdout:+ Name: "smithi166-lv-1", 2022-03-07T23:37:09.942 INFO:tasks.rook.operator.smithi078.stdout:+ Count: 1, 2022-03-07T23:37:09.942 INFO:tasks.rook.operator.smithi078.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-07T23:37:09.942 INFO:tasks.rook.operator.smithi078.stdout:+ }, 2022-03-07T23:37:09.942 INFO:tasks.rook.operator.smithi078.stdout:+ { 2022-03-07T23:37:09.942 INFO:tasks.rook.operator.smithi078.stdout:+ Name: "smithi166-lv-2", 2022-03-07T23:37:09.943 INFO:tasks.rook.operator.smithi078.stdout:+ Count: 1, 2022-03-07T23:37:09.943 INFO:tasks.rook.operator.smithi078.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-07T23:37:09.943 INFO:tasks.rook.operator.smithi078.stdout:+ }, 2022-03-07T23:37:09.943 INFO:tasks.rook.operator.smithi078.stdout:+ { 2022-03-07T23:37:09.944 INFO:tasks.rook.operator.smithi078.stdout:+ Name: "smithi166-lv-3", 2022-03-07T23:37:09.944 INFO:tasks.rook.operator.smithi078.stdout:+ Count: 1, 2022-03-07T23:37:09.944 INFO:tasks.rook.operator.smithi078.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-07T23:37:09.944 INFO:tasks.rook.operator.smithi078.stdout:+ }, 2022-03-07T23:37:09.944 INFO:tasks.rook.operator.smithi078.stdout:+ { 2022-03-07T23:37:09.945 INFO:tasks.rook.operator.smithi078.stdout:+ Name: "smithi166-lv-4", 2022-03-07T23:37:09.945 INFO:tasks.rook.operator.smithi078.stdout:+ Count: 1, 2022-03-07T23:37:09.945 INFO:tasks.rook.operator.smithi078.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-07T23:37:09.945 INFO:tasks.rook.operator.smithi078.stdout:+ }, 2022-03-07T23:37:09.945 INFO:tasks.rook.operator.smithi078.stdout:+ { 2022-03-07T23:37:09.946 INFO:tasks.rook.operator.smithi078.stdout:+ Name: "smithi170-lv-1", 2022-03-07T23:37:09.946 INFO:tasks.rook.operator.smithi078.stdout:+ Count: 1, 2022-03-07T23:37:09.946 INFO:tasks.rook.operator.smithi078.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-07T23:37:09.946 INFO:tasks.rook.operator.smithi078.stdout:+ }, 2022-03-07T23:37:09.947 INFO:tasks.rook.operator.smithi078.stdout:+ { 2022-03-07T23:37:09.947 INFO:tasks.rook.operator.smithi078.stdout:+ Name: "smithi170-lv-2", 2022-03-07T23:37:09.947 INFO:tasks.rook.operator.smithi078.stdout:+ Count: 1, 2022-03-07T23:37:09.947 INFO:tasks.rook.operator.smithi078.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-07T23:37:09.947 INFO:tasks.rook.operator.smithi078.stdout:+ }, 2022-03-07T23:37:09.948 INFO:tasks.rook.operator.smithi078.stdout:+ { 2022-03-07T23:37:09.948 INFO:tasks.rook.operator.smithi078.stdout:+ Name: "smithi170-lv-3", 2022-03-07T23:37:09.948 INFO:tasks.rook.operator.smithi078.stdout:+ Count: 1, 2022-03-07T23:37:09.948 INFO:tasks.rook.operator.smithi078.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-07T23:37:09.949 INFO:tasks.rook.operator.smithi078.stdout:+ }, 2022-03-07T23:37:09.949 INFO:tasks.rook.operator.smithi078.stdout:+ { 2022-03-07T23:37:09.949 INFO:tasks.rook.operator.smithi078.stdout:+ Name: "smithi170-lv-4", 2022-03-07T23:37:09.949 INFO:tasks.rook.operator.smithi078.stdout:+ Count: 1, 2022-03-07T23:37:09.949 INFO:tasks.rook.operator.smithi078.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{...}}, 2022-03-07T23:37:09.950 INFO:tasks.rook.operator.smithi078.stdout:+ }, 2022-03-07T23:37:09.950 INFO:tasks.rook.operator.smithi078.stdout:+ }, 2022-03-07T23:37:09.950 INFO:tasks.rook.operator.smithi078.stdout: }, 2022-03-07T23:37:09.950 INFO:tasks.rook.operator.smithi078.stdout: Annotations: nil, 2022-03-07T23:37:09.950 INFO:tasks.rook.operator.smithi078.stdout: Labels: nil, 2022-03-07T23:37:09.951 INFO:tasks.rook.operator.smithi078.stdout: ... // 20 identical fields 2022-03-07T23:37:09.951 INFO:tasks.rook.operator.smithi078.stdout: } 2022-03-07T23:37:09.951 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:09.931822 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-07T23:37:09.951 INFO:tasks.rook.operator.smithi078.stdout:I0307 23:37:09.932338 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-07T23:37:09.952 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:09.932413 I | op-mon: stopping monitoring of mons in namespace "rook-ceph" 2022-03-07T23:37:09.952 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:09.932443 I | op-osd: stopping monitoring of OSDs in namespace "rook-ceph" 2022-03-07T23:37:09.952 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:09.932458 I | ceph-cluster-controller: stopping monitoring of ceph status 2022-03-07T23:37:09.952 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:09.932472 I | operator: successfully started the controller-runtime manager 2022-03-07T23:37:09.952 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:09.938617 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-07T23:37:09.953 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:09.938644 I | operator: watching all namespaces for Ceph CRs 2022-03-07T23:37:09.953 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:09.938674 I | operator: setting up schemes 2022-03-07T23:37:09.953 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:09.941432 I | operator: setting up the controller-runtime manager 2022-03-07T23:37:09.954 INFO:tasks.rook:Waiting for 12 OSDs 2022-03-07T23:37:09.955 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd stat -f json 2022-03-07T23:37:10.141 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.145318 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-07T23:37:10.147 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.147682 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-07T23:37:10.147 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.147727 I | ceph-cluster-controller: successfully started 2022-03-07T23:37:10.147 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.147788 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-07T23:37:10.148 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.147812 I | ceph-crashcollector-controller: successfully started 2022-03-07T23:37:10.148 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.147837 I | ceph-block-pool-controller: successfully started 2022-03-07T23:37:10.148 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.147868 I | ceph-object-store-user-controller: successfully started 2022-03-07T23:37:10.149 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.147890 I | ceph-object-realm-controller: successfully started 2022-03-07T23:37:10.149 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.147902 I | ceph-object-zonegroup-controller: successfully started 2022-03-07T23:37:10.150 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.147914 I | ceph-object-zone-controller: successfully started 2022-03-07T23:37:10.150 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.148125 I | ceph-object-controller: successfully started 2022-03-07T23:37:10.150 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.148167 I | ceph-file-controller: successfully started 2022-03-07T23:37:10.150 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.148220 I | ceph-nfs-controller: successfully started 2022-03-07T23:37:10.150 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.148258 I | ceph-rbd-mirror-controller: successfully started 2022-03-07T23:37:10.151 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.148301 I | ceph-client-controller: successfully started 2022-03-07T23:37:10.151 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.148322 I | ceph-filesystem-mirror-controller: successfully started 2022-03-07T23:37:10.151 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.148355 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-07T23:37:10.151 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.148378 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-07T23:37:10.152 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.148399 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-07T23:37:10.152 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.148417 I | ceph-bucket-topic: successfully started 2022-03-07T23:37:10.152 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.148440 I | ceph-bucket-notification: successfully started 2022-03-07T23:37:10.152 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.148456 I | ceph-bucket-notification: successfully started 2022-03-07T23:37:10.152 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.148476 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-07T23:37:10.153 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.149585 I | operator: starting the controller-runtime manager 2022-03-07T23:37:10.251 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.255139 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-07T23:37:10.252 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.255189 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-07T23:37:10.252 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.255210 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-07T23:37:10.252 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.255386 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-07T23:37:10.258 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.260207 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-07T23:37:10.259 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.260503 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:37:10.259 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.260557 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-07T23:37:10.259 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.260570 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-07T23:37:10.260 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.261313 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:37:10.261 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.261857 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-07T23:37:10.261 INFO:tasks.rook.operator.smithi078.stdout:I0307 23:37:10.261925 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-07T23:37:10.273 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.277096 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-07T23:37:10.277 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.279729 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-07T23:37:10.298 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.299442 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-07T23:37:10.298 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.299469 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-07T23:37:10.298 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.299476 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-07T23:37:10.299 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.299481 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-07T23:37:10.299 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.299487 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-07T23:37:10.300 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.299494 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-07T23:37:10.300 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.299500 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-07T23:37:10.300 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.299509 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-07T23:37:10.300 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.299520 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-07T23:37:10.301 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.299531 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-07T23:37:10.301 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.299540 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-07T23:37:10.301 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.299551 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-07T23:37:10.301 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.299565 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-07T23:37:10.301 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.299573 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-07T23:37:10.302 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.299583 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-07T23:37:10.302 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.299594 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-07T23:37:10.302 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.299654 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-07T23:37:10.302 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:10.299671 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-07T23:37:10.580 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:37:10.606 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-07T23:37:10.607 INFO:tasks.rook: have 0/12 OSDs 2022-03-07T23:37:17.878 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.880155 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-07T23:37:17.878 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.880194 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-07T23:37:17.879 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.880209 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-07T23:37:17.879 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.880233 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-07T23:37:17.879 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.880246 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-07T23:37:17.879 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.880254 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-07T23:37:17.880 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.880275 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-07T23:37:17.880 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.880289 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-07T23:37:17.880 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.880303 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-07T23:37:17.880 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.880319 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-07T23:37:17.881 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.880332 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-07T23:37:17.881 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.880342 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-07T23:37:17.881 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.880353 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-07T23:37:17.881 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.880363 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-07T23:37:17.881 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.880374 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-07T23:37:17.882 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.880386 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-07T23:37:17.882 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.880398 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-07T23:37:17.882 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.880409 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-07T23:37:17.882 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.880420 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-07T23:37:17.882 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.880431 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-07T23:37:17.883 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.880444 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-07T23:37:17.883 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.880454 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-07T23:37:17.883 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.880467 I | ceph-csi: Kubernetes version is 1.23 2022-03-07T23:37:17.883 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.880480 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-07T23:37:17.884 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.880489 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-07T23:37:17.927 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.931421 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-07T23:37:17.936 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.939484 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-07T23:37:17.937 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.939504 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-07T23:37:17.937 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.939508 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-07T23:37:17.937 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.939510 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-07T23:37:17.937 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.939513 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-07T23:37:17.938 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.939516 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-07T23:37:17.938 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:17.939519 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-07T23:37:18.076 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:18.079826 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-07T23:37:18.076 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:18.079846 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-07T23:37:18.077 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:18.079852 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-07T23:37:18.081 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:18.081393 I | ceph-spec: detected ceph image version: "17.0.0-11002 quincy" 2022-03-07T23:37:18.081 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:18.081412 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-07T23:37:18.082 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:18.081422 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11002 quincy", pursuing 2022-03-07T23:37:18.083 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:18.086774 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:37:18.187 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:18.191430 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T23:37:18.188 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:18.191692 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T23:37:18.195 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:18.198871 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-07T23:37:18.408 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:18.411953 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-07T23:37:18.408 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:18.411977 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-07T23:37:18.408 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:18.411983 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-07T23:37:18.453 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:18.457165 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-07T23:37:18.453 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:18.457187 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-07T23:37:18.454 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:18.457194 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-07T23:37:18.528 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:18.532214 I | ceph-csi: successfully started CSI CephFS driver 2022-03-07T23:37:18.639 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:18.643189 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11002 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4" 2022-03-07T23:37:18.710 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:18.712036 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-07T23:37:18.713 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:18.717420 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-07T23:37:18.713 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:18.717434 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-07T23:37:18.760 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:18.764622 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-07T23:37:18.812 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:18.816564 I | op-mon: start running mons 2022-03-07T23:37:19.182 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:19.186040 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:37:20.183 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:20.187243 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.105.146.196:6789","10.98.222.16:6789","10.96.111.213:6789"]}] data:a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 mapping:{"node":{"a":{"Name":"smithi170","Hostname":"smithi170","Address":"172.21.15.170"},"b":{"Name":"smithi166","Hostname":"smithi166","Address":"172.21.15.166"},"c":{"Name":"smithi078","Hostname":"smithi078","Address":"172.21.15.78"}}} maxMonId:2] 2022-03-07T23:37:20.608 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd stat -f json 2022-03-07T23:37:20.782 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:20.786369 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T23:37:20.783 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:20.786688 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T23:37:21.133 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:37:21.161 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-07T23:37:21.162 INFO:tasks.rook: have 0/12 OSDs 2022-03-07T23:37:22.381 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:22.385693 I | op-mon: targeting the mon count 3 2022-03-07T23:37:22.387 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:22.391618 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T23:37:22.791 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:22.795564 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T23:37:22.792 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:22.795593 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T23:37:23.134 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:23.134042 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T23:37:23.135 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:23.134073 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T23:37:23.467 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:23.471501 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T23:37:23.468 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:23.471524 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-07T23:37:23.891 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:23.894658 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-07T23:37:23.891 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:23.894684 I | op-config: deleting "log file" option from the mon configuration database 2022-03-07T23:37:24.230 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:24.234007 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-07T23:37:24.273 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:24.234037 I | op-mon: checking for basic quorum with existing mons 2022-03-07T23:37:24.345 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:24.349633 I | op-mon: mon "a" endpoint is [v2:10.105.146.196:3300,v1:10.105.146.196:6789] 2022-03-07T23:37:24.483 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:24.481135 I | op-mon: mon "b" endpoint is [v2:10.98.222.16:3300,v1:10.98.222.16:6789] 2022-03-07T23:37:24.783 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:24.787801 I | op-mon: mon "c" endpoint is [v2:10.96.111.213:3300,v1:10.96.111.213:6789] 2022-03-07T23:37:25.383 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:25.387050 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.105.146.196:6789","10.98.222.16:6789","10.96.111.213:6789"]}] data:a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 mapping:{"node":{"a":{"Name":"smithi170","Hostname":"smithi170","Address":"172.21.15.170"},"b":{"Name":"smithi166","Hostname":"smithi166","Address":"172.21.15.166"},"c":{"Name":"smithi078","Hostname":"smithi078","Address":"172.21.15.78"}}} maxMonId:2] 2022-03-07T23:37:25.983 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:25.986802 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T23:37:25.987 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:25.987109 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T23:37:26.401 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:26.404915 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-07T23:37:26.416 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:26.420268 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-07T23:37:26.416 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:26.420303 I | op-mon: waiting for mon quorum with [a b c] 2022-03-07T23:37:26.987 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:26.991018 I | op-mon: mons running: [a b c] 2022-03-07T23:37:27.451 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:27.454320 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T23:37:27.457 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:27.461399 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-07T23:37:27.468 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:27.472005 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-07T23:37:27.468 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:27.472033 I | op-mon: waiting for mon quorum with [a b c] 2022-03-07T23:37:27.590 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:27.594320 I | op-mon: mons running: [a b c] 2022-03-07T23:37:28.009 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:28.013565 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T23:37:28.016 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:28.021102 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-07T23:37:28.028 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:28.032941 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-07T23:37:28.029 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:28.032973 I | op-mon: waiting for mon quorum with [a b c] 2022-03-07T23:37:28.191 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:28.195309 I | op-mon: mons running: [a b c] 2022-03-07T23:37:28.601 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:28.605446 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T23:37:28.601 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:28.605479 I | op-mon: mons created: 3 2022-03-07T23:37:29.011 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:29.015541 I | op-mon: waiting for mon quorum with [a b c] 2022-03-07T23:37:29.034 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:29.038803 I | op-mon: mons running: [a b c] 2022-03-07T23:37:29.459 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:29.463719 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T23:37:29.460 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:29.463767 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-07T23:37:29.894 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:29.898014 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-07T23:37:30.370 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:30.374692 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-07T23:37:30.871 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:30.874895 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-07T23:37:31.163 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd stat -f json 2022-03-07T23:37:31.314 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:31.317912 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-07T23:37:31.314 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:31.317937 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-07T23:37:31.663 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:37:31.682 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-07T23:37:31.683 INFO:tasks.rook: have 0/12 OSDs 2022-03-07T23:37:31.757 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:31.761081 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-07T23:37:32.167 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:32.171526 I | cephclient: successfully enabled msgr2 protocol 2022-03-07T23:37:32.168 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:32.171562 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-07T23:37:32.168 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:32.171570 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-07T23:37:32.593 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:32.597340 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-07T23:37:32.658 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:32.662917 I | op-mgr: start running mgr 2022-03-07T23:37:32.659 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:32.662951 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-07T23:37:33.274 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:33.278994 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-07T23:37:33.292 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:33.296869 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-07T23:37:33.297 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:33.300023 I | op-mgr: setting services to point to mgr "a" 2022-03-07T23:37:33.437 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:33.441423 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-07T23:37:33.438 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:33.441537 I | op-mgr: successful modules: balancer 2022-03-07T23:37:33.447 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:33.451207 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-07T23:37:33.447 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:33.451259 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-07T23:37:33.468 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:33.472137 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-07T23:37:33.496 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:33.500432 I | op-osd: creating 1 new PVCs for device set "smithi078-lv-1" 2022-03-07T23:37:33.700 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:33.704135 I | op-osd: successfully provisioned PVC "smithi078-lv-1-data-0mvnbx" 2022-03-07T23:37:33.700 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:33.704162 I | op-osd: creating 1 new PVCs for device set "smithi078-lv-2" 2022-03-07T23:37:33.897 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:33.901516 I | op-osd: successfully provisioned PVC "smithi078-lv-2-data-07kjmr" 2022-03-07T23:37:33.897 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:33.901552 I | op-osd: creating 1 new PVCs for device set "smithi078-lv-3" 2022-03-07T23:37:34.097 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:34.101776 I | op-osd: successfully provisioned PVC "smithi078-lv-3-data-0wsgp2" 2022-03-07T23:37:34.098 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:34.101810 I | op-osd: creating 1 new PVCs for device set "smithi078-lv-4" 2022-03-07T23:37:34.297 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:34.301667 I | op-osd: successfully provisioned PVC "smithi078-lv-4-data-0mvh7t" 2022-03-07T23:37:34.298 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:34.301708 I | op-osd: creating 1 new PVCs for device set "smithi166-lv-1" 2022-03-07T23:37:34.347 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:34.350883 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-07T23:37:34.371 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:34.375729 I | op-mgr: successful modules: prometheus 2022-03-07T23:37:34.373 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:34.376126 I | op-mgr: successful modules: dashboard 2022-03-07T23:37:34.498 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:34.502615 I | op-osd: successfully provisioned PVC "smithi166-lv-1-data-09wrcv" 2022-03-07T23:37:34.499 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:34.502647 I | op-osd: creating 1 new PVCs for device set "smithi166-lv-2" 2022-03-07T23:37:34.727 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:34.731734 I | op-osd: successfully provisioned PVC "smithi166-lv-2-data-0tz6t9" 2022-03-07T23:37:34.728 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:34.731769 I | op-osd: creating 1 new PVCs for device set "smithi166-lv-3" 2022-03-07T23:37:34.897 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:34.902002 I | op-osd: successfully provisioned PVC "smithi166-lv-3-data-02vsxz" 2022-03-07T23:37:34.898 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:34.902036 I | op-osd: creating 1 new PVCs for device set "smithi166-lv-4" 2022-03-07T23:37:35.098 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:35.102137 I | op-osd: successfully provisioned PVC "smithi166-lv-4-data-0cg6jn" 2022-03-07T23:37:35.098 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:35.102173 I | op-osd: creating 1 new PVCs for device set "smithi170-lv-1" 2022-03-07T23:37:35.361 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:35.365518 I | op-osd: successfully provisioned PVC "smithi170-lv-1-data-076jz6" 2022-03-07T23:37:35.362 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:35.365561 I | op-osd: creating 1 new PVCs for device set "smithi170-lv-2" 2022-03-07T23:37:35.503 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:35.507775 I | op-osd: successfully provisioned PVC "smithi170-lv-2-data-0w6nq6" 2022-03-07T23:37:35.504 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:35.507809 I | op-osd: creating 1 new PVCs for device set "smithi170-lv-3" 2022-03-07T23:37:35.698 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:35.702118 I | op-osd: successfully provisioned PVC "smithi170-lv-3-data-0rc4dr" 2022-03-07T23:37:35.698 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:35.702156 I | op-osd: creating 1 new PVCs for device set "smithi170-lv-4" 2022-03-07T23:37:35.898 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:35.902322 I | op-osd: successfully provisioned PVC "smithi170-lv-4-data-0lsgm7" 2022-03-07T23:37:35.906 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:35.910226 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-1-data-0mvnbx" 2022-03-07T23:37:35.993 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:35.997379 I | op-mgr: successful modules: orchestrator modules 2022-03-07T23:37:36.360 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:36.364637 I | op-osd: started OSD provisioning job for PVC "smithi078-lv-1-data-0mvnbx" 2022-03-07T23:37:36.361 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:36.364672 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-2-data-07kjmr" 2022-03-07T23:37:36.709 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:36.713463 I | op-osd: started OSD provisioning job for PVC "smithi078-lv-2-data-07kjmr" 2022-03-07T23:37:36.709 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:36.713500 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-3-data-0wsgp2" 2022-03-07T23:37:37.109 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:37.113810 I | op-osd: started OSD provisioning job for PVC "smithi078-lv-3-data-0wsgp2" 2022-03-07T23:37:37.109 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:37.113846 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-4-data-0mvh7t" 2022-03-07T23:37:38.022 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:38.026915 I | op-osd: started OSD provisioning job for PVC "smithi078-lv-4-data-0mvh7t" 2022-03-07T23:37:38.023 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:38.026952 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-1-data-09wrcv" 2022-03-07T23:37:38.407 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:38.411605 I | op-osd: started OSD provisioning job for PVC "smithi166-lv-1-data-09wrcv" 2022-03-07T23:37:38.408 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:38.411637 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-2-data-0tz6t9" 2022-03-07T23:37:38.617 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:38.621371 I | op-osd: started OSD provisioning job for PVC "smithi166-lv-2-data-0tz6t9" 2022-03-07T23:37:38.617 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:38.621405 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-3-data-02vsxz" 2022-03-07T23:37:38.899 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:38.903278 I | op-osd: started OSD provisioning job for PVC "smithi166-lv-3-data-02vsxz" 2022-03-07T23:37:38.899 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:38.903314 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-4-data-0cg6jn" 2022-03-07T23:37:39.112 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:39.116941 I | op-osd: started OSD provisioning job for PVC "smithi166-lv-4-data-0cg6jn" 2022-03-07T23:37:39.113 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:39.116963 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-1-data-076jz6" 2022-03-07T23:37:39.514 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:39.518927 I | op-osd: started OSD provisioning job for PVC "smithi170-lv-1-data-076jz6" 2022-03-07T23:37:39.515 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:39.518952 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-2-data-0w6nq6" 2022-03-07T23:37:40.310 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:40.314038 I | op-osd: started OSD provisioning job for PVC "smithi170-lv-2-data-0w6nq6" 2022-03-07T23:37:40.310 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:40.314072 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-3-data-0rc4dr" 2022-03-07T23:37:40.819 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:40.823266 I | op-osd: started OSD provisioning job for PVC "smithi170-lv-3-data-0rc4dr" 2022-03-07T23:37:40.819 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:40.823288 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-4-data-0lsgm7" 2022-03-07T23:37:40.977 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:40.981774 I | op-osd: started OSD provisioning job for PVC "smithi170-lv-4-data-0lsgm7" 2022-03-07T23:37:40.978 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:40.981798 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-07T23:37:40.978 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:40.981802 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-07T23:37:40.998 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:41.002072 I | op-osd: OSD orchestration status for PVC smithi078-lv-1-data-0mvnbx is "starting" 2022-03-07T23:37:40.998 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:41.002105 I | op-osd: OSD orchestration status for PVC smithi078-lv-2-data-07kjmr is "starting" 2022-03-07T23:37:40.998 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:41.002119 I | op-osd: OSD orchestration status for PVC smithi078-lv-3-data-0wsgp2 is "starting" 2022-03-07T23:37:40.999 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:41.002130 I | op-osd: OSD orchestration status for PVC smithi078-lv-4-data-0mvh7t is "starting" 2022-03-07T23:37:40.999 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:41.002141 I | op-osd: OSD orchestration status for PVC smithi166-lv-1-data-09wrcv is "starting" 2022-03-07T23:37:40.999 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:41.002152 I | op-osd: OSD orchestration status for PVC smithi166-lv-2-data-0tz6t9 is "starting" 2022-03-07T23:37:40.999 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:41.002162 I | op-osd: OSD orchestration status for PVC smithi166-lv-3-data-02vsxz is "starting" 2022-03-07T23:37:41.000 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:41.002173 I | op-osd: OSD orchestration status for PVC smithi166-lv-4-data-0cg6jn is "starting" 2022-03-07T23:37:41.000 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:41.002184 I | op-osd: OSD orchestration status for PVC smithi170-lv-1-data-076jz6 is "starting" 2022-03-07T23:37:41.001 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:41.002196 I | op-osd: OSD orchestration status for PVC smithi170-lv-2-data-0w6nq6 is "starting" 2022-03-07T23:37:41.001 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:41.002207 I | op-osd: OSD orchestration status for PVC smithi170-lv-3-data-0rc4dr is "starting" 2022-03-07T23:37:41.001 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:41.002218 I | op-osd: OSD orchestration status for PVC smithi170-lv-4-data-0lsgm7 is "starting" 2022-03-07T23:37:41.685 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd stat -f json 2022-03-07T23:37:42.583 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:37:42.606 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-03-07T23:37:42.606 INFO:tasks.rook: have 0/12 OSDs 2022-03-07T23:37:46.697 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:46.701936 I | op-osd: OSD orchestration status for node smithi078-lv-2-data-07kjmr is "orchestrating" 2022-03-07T23:37:47.060 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:47.064581 I | op-osd: OSD orchestration status for PVC smithi078-lv-2-data-07kjmr is "orchestrating" 2022-03-07T23:37:48.841 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:48.846388 I | op-osd: OSD orchestration status for node smithi166-lv-1-data-09wrcv is "orchestrating" 2022-03-07T23:37:49.220 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:49.225076 I | op-osd: OSD orchestration status for PVC smithi166-lv-1-data-09wrcv is "orchestrating" 2022-03-07T23:37:50.318 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:50.322763 I | op-osd: OSD orchestration status for node smithi166-lv-2-data-0tz6t9 is "orchestrating" 2022-03-07T23:37:50.766 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:50.768550 I | op-osd: OSD orchestration status for PVC smithi166-lv-2-data-0tz6t9 is "orchestrating" 2022-03-07T23:37:52.608 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd stat -f json 2022-03-07T23:37:52.665 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:52.670554 I | op-osd: OSD orchestration status for node smithi170-lv-1-data-076jz6 is "orchestrating" 2022-03-07T23:37:53.209 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:53.213160 I | op-osd: OSD orchestration status for PVC smithi170-lv-1-data-076jz6 is "orchestrating" 2022-03-07T23:37:53.310 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:37:53.332 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":4,"num_osds":2,"num_up_osds":0,"osd_up_since":0,"num_in_osds":2,"osd_in_since":1646696271,"num_remapped_pgs":0} 2022-03-07T23:37:53.332 INFO:tasks.rook: have 0/12 OSDs 2022-03-07T23:37:54.282 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:54.287346 I | op-osd: OSD orchestration status for PVC smithi078-lv-2-data-07kjmr is "completed" 2022-03-07T23:37:54.283 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:54.287394 I | op-osd: creating OSD 0 on PVC "smithi078-lv-2-data-07kjmr" 2022-03-07T23:37:54.283 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:54.287428 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-2-data-07kjmr" 2022-03-07T23:37:57.087 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:57.090197 I | op-osd: OSD orchestration status for node smithi170-lv-2-data-0w6nq6 is "orchestrating" 2022-03-07T23:37:57.573 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:57.578538 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi078": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T23:37:58.059 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:58.064243 I | op-osd: OSD orchestration status for PVC smithi166-lv-1-data-09wrcv is "completed" 2022-03-07T23:37:58.060 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:58.064276 I | op-osd: creating OSD 1 on PVC "smithi166-lv-1-data-09wrcv" 2022-03-07T23:37:58.060 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:58.064292 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-1-data-09wrcv" 2022-03-07T23:37:59.579 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:37:59.581795 I | op-osd: OSD orchestration status for PVC smithi170-lv-2-data-0w6nq6 is "orchestrating" 2022-03-07T23:38:01.227 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:01.231539 I | op-osd: OSD orchestration status for PVC smithi170-lv-1-data-076jz6 is "completed" 2022-03-07T23:38:01.227 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:01.231580 I | op-osd: creating OSD 2 on PVC "smithi170-lv-1-data-076jz6" 2022-03-07T23:38:01.227 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:01.231600 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-1-data-076jz6" 2022-03-07T23:38:02.950 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:02.954507 I | op-osd: OSD orchestration status for node smithi170-lv-4-data-0lsgm7 is "orchestrating" 2022-03-07T23:38:03.126 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:03.130968 I | op-osd: OSD orchestration status for node smithi166-lv-4-data-0cg6jn is "orchestrating" 2022-03-07T23:38:03.126 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:03.131147 I | op-osd: OSD orchestration status for node smithi166-lv-3-data-02vsxz is "orchestrating" 2022-03-07T23:38:03.334 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd stat -f json 2022-03-07T23:38:03.631 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:03.634861 I | op-osd: OSD orchestration status for PVC smithi166-lv-2-data-0tz6t9 is "completed" 2022-03-07T23:38:03.631 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:03.634880 I | op-osd: creating OSD 3 on PVC "smithi166-lv-2-data-0tz6t9" 2022-03-07T23:38:03.632 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:03.634888 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-2-data-0tz6t9" 2022-03-07T23:38:04.850 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:38:04.876 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":7,"num_osds":5,"num_up_osds":0,"osd_up_since":0,"num_in_osds":5,"osd_in_since":1646696280,"num_remapped_pgs":0} 2022-03-07T23:38:04.877 INFO:tasks.rook: have 0/12 OSDs 2022-03-07T23:38:05.123 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:05.128404 I | op-osd: OSD orchestration status for PVC smithi170-lv-4-data-0lsgm7 is "orchestrating" 2022-03-07T23:38:05.128 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:05.128652 I | op-osd: OSD orchestration status for PVC smithi166-lv-3-data-02vsxz is "orchestrating" 2022-03-07T23:38:05.129 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:05.128679 I | op-osd: OSD orchestration status for PVC smithi166-lv-4-data-0cg6jn is "orchestrating" 2022-03-07T23:38:06.173 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:06.178509 I | op-osd: OSD orchestration status for PVC smithi170-lv-2-data-0w6nq6 is "completed" 2022-03-07T23:38:06.174 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:06.178543 I | op-osd: creating OSD 4 on PVC "smithi170-lv-2-data-0w6nq6" 2022-03-07T23:38:06.174 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:06.178564 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-2-data-0w6nq6" 2022-03-07T23:38:07.637 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:07.641897 I | op-osd: OSD orchestration status for node smithi170-lv-3-data-0rc4dr is "orchestrating" 2022-03-07T23:38:08.524 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:08.528670 I | op-osd: OSD orchestration status for node smithi078-lv-1-data-0mvnbx is "orchestrating" 2022-03-07T23:38:08.550 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:08.554807 I | op-osd: OSD orchestration status for PVC smithi170-lv-3-data-0rc4dr is "orchestrating" 2022-03-07T23:38:09.455 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:09.460277 I | op-osd: OSD orchestration status for PVC smithi078-lv-1-data-0mvnbx is "orchestrating" 2022-03-07T23:38:10.970 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:10.975107 I | op-osd: OSD orchestration status for node smithi078-lv-3-data-0wsgp2 is "orchestrating" 2022-03-07T23:38:11.021 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:11.026352 I | op-osd: OSD orchestration status for node smithi078-lv-4-data-0mvh7t is "orchestrating" 2022-03-07T23:38:11.942 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:11.947845 I | op-osd: OSD orchestration status for PVC smithi078-lv-3-data-0wsgp2 is "orchestrating" 2022-03-07T23:38:11.943 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:11.948035 I | op-osd: OSD orchestration status for PVC smithi078-lv-4-data-0mvh7t is "orchestrating" 2022-03-07T23:38:12.300 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:12.305544 I | op-osd: OSD orchestration status for PVC smithi166-lv-3-data-02vsxz is "completed" 2022-03-07T23:38:12.301 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:12.305570 I | op-osd: creating OSD 5 on PVC "smithi166-lv-3-data-02vsxz" 2022-03-07T23:38:12.301 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:12.305579 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-3-data-02vsxz" 2022-03-07T23:38:12.630 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:12.635890 I | op-osd: OSD orchestration status for PVC smithi170-lv-4-data-0lsgm7 is "completed" 2022-03-07T23:38:12.631 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:12.635914 I | op-osd: creating OSD 6 on PVC "smithi170-lv-4-data-0lsgm7" 2022-03-07T23:38:12.631 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:12.635924 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-4-data-0lsgm7" 2022-03-07T23:38:14.342 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:14.347159 I | op-osd: OSD orchestration status for PVC smithi166-lv-4-data-0cg6jn is "completed" 2022-03-07T23:38:14.342 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:14.347189 I | op-osd: creating OSD 7 on PVC "smithi166-lv-4-data-0cg6jn" 2022-03-07T23:38:14.343 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:14.347201 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-4-data-0cg6jn" 2022-03-07T23:38:14.878 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd stat -f json 2022-03-07T23:38:16.321 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:16.326452 I | op-osd: OSD orchestration status for PVC smithi170-lv-3-data-0rc4dr is "completed" 2022-03-07T23:38:16.322 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:16.326477 I | op-osd: creating OSD 8 on PVC "smithi170-lv-3-data-0rc4dr" 2022-03-07T23:38:16.322 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:16.326488 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-3-data-0rc4dr" 2022-03-07T23:38:17.136 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:38:17.159 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":14,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1646696294,"num_remapped_pgs":0} 2022-03-07T23:38:17.159 INFO:tasks.rook: have 0/12 OSDs 2022-03-07T23:38:17.867 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:17.873023 I | op-osd: OSD orchestration status for PVC smithi078-lv-1-data-0mvnbx is "completed" 2022-03-07T23:38:17.868 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:17.873050 I | op-osd: creating OSD 9 on PVC "smithi078-lv-1-data-0mvnbx" 2022-03-07T23:38:17.868 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:17.873059 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-1-data-0mvnbx" 2022-03-07T23:38:20.814 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:20.819595 I | op-osd: OSD orchestration status for PVC smithi078-lv-4-data-0mvh7t is "completed" 2022-03-07T23:38:20.815 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:20.819658 I | op-osd: creating OSD 10 on PVC "smithi078-lv-4-data-0mvh7t" 2022-03-07T23:38:20.815 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:20.819688 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-4-data-0mvh7t" 2022-03-07T23:38:22.791 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:22.794364 I | op-osd: OSD orchestration status for PVC smithi078-lv-3-data-0wsgp2 is "completed" 2022-03-07T23:38:22.791 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:22.794389 I | op-osd: creating OSD 11 on PVC "smithi078-lv-3-data-0wsgp2" 2022-03-07T23:38:22.792 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:22.794400 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-3-data-0wsgp2" 2022-03-07T23:38:24.525 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:24.530365 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-07T23:38:24.526 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:38:24.530396 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-07T23:38:27.160 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd stat -f json 2022-03-07T23:38:28.596 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:38:28.620 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":14,"num_osds":12,"num_up_osds":0,"osd_up_since":0,"num_in_osds":12,"osd_in_since":1646696294,"num_remapped_pgs":0} 2022-03-07T23:38:28.620 INFO:tasks.rook: have 0/12 OSDs 2022-03-07T23:38:38.621 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd stat -f json 2022-03-07T23:38:39.456 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:38:39.475 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":21,"num_osds":12,"num_up_osds":4,"osd_up_since":1646696319,"num_in_osds":12,"osd_in_since":1646696294,"num_remapped_pgs":0} 2022-03-07T23:38:39.476 INFO:tasks.rook: have 4/12 OSDs 2022-03-07T23:38:49.477 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd stat -f json 2022-03-07T23:38:50.194 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:38:50.219 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":29,"num_osds":12,"num_up_osds":9,"osd_up_since":1646696326,"num_in_osds":12,"osd_in_since":1646696294,"num_remapped_pgs":0} 2022-03-07T23:38:50.220 INFO:tasks.rook: have 9/12 OSDs 2022-03-07T23:39:00.221 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd stat -f json 2022-03-07T23:39:00.834 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:39:00.856 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":35,"num_osds":12,"num_up_osds":12,"osd_up_since":1646696335,"num_in_osds":12,"osd_in_since":1646696294,"num_remapped_pgs":0} 2022-03-07T23:39:00.857 INFO:tasks.rook:Distributing ceph config and client.admin keyring 2022-03-07T23:39:00.857 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- cat /etc/ceph/ceph.conf 2022-03-07T23:39:01.005 INFO:teuthology.orchestra.run.smithi078.stdout:[global] 2022-03-07T23:39:01.005 INFO:teuthology.orchestra.run.smithi078.stdout:mon_host = 10.105.146.196:6789,10.98.222.16:6789,10.96.111.213:6789 2022-03-07T23:39:01.006 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:39:01.006 INFO:teuthology.orchestra.run.smithi078.stdout:[client.admin] 2022-03-07T23:39:01.006 INFO:teuthology.orchestra.run.smithi078.stdout:keyring = /etc/ceph/keyring 2022-03-07T23:39:01.010 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- cat /etc/ceph/keyring 2022-03-07T23:39:01.207 INFO:teuthology.orchestra.run.smithi078.stdout:[client.admin] 2022-03-07T23:39:01.207 INFO:teuthology.orchestra.run.smithi078.stdout:key = AQBmliZiGKlXARAAZQQrrxOGTff39ZwmK04syA== 2022-03-07T23:39:01.211 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /etc/ceph 2022-03-07T23:39:01.515 DEBUG:teuthology.orchestra.run.smithi166:> sudo mkdir -p /etc/ceph 2022-03-07T23:39:01.532 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /etc/ceph 2022-03-07T23:39:01.548 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2022-03-07T23:39:01.548 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.conf 2022-03-07T23:39:01.568 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2022-03-07T23:39:01.568 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/keyring 2022-03-07T23:39:01.580 DEBUG:teuthology.orchestra.run.smithi166:> set -ex 2022-03-07T23:39:01.581 DEBUG:teuthology.orchestra.run.smithi166:> sudo dd of=/etc/ceph/ceph.conf 2022-03-07T23:39:01.598 DEBUG:teuthology.orchestra.run.smithi166:> set -ex 2022-03-07T23:39:01.598 DEBUG:teuthology.orchestra.run.smithi166:> sudo dd of=/etc/ceph/keyring 2022-03-07T23:39:01.614 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2022-03-07T23:39:01.614 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.conf 2022-03-07T23:39:01.630 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2022-03-07T23:39:01.631 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/keyring 2022-03-07T23:39:01.642 INFO:tasks.rook:Setting up client nodes... 2022-03-07T23:39:01.643 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph auth get-or-create client.a mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-07T23:39:02.353 INFO:teuthology.orchestra.run.smithi078.stdout:[client.a] 2022-03-07T23:39:02.354 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCWlyZiy/78DRAAZdEtKi7aos3HFawIAJOuxg== 2022-03-07T23:39:02.375 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2022-03-07T23:39:02.376 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.client.a.keyring 2022-03-07T23:39:02.376 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/ceph.client.a.keyring 2022-03-07T23:39:02.401 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph auth get-or-create client.b mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-07T23:39:03.146 INFO:teuthology.orchestra.run.smithi078.stdout:[client.b] 2022-03-07T23:39:03.146 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCXlyZi77zNARAAL6fWGgOxfzyEasqR0oPFgg== 2022-03-07T23:39:03.169 DEBUG:teuthology.orchestra.run.smithi166:> set -ex 2022-03-07T23:39:03.170 DEBUG:teuthology.orchestra.run.smithi166:> sudo dd of=/etc/ceph/ceph.client.b.keyring 2022-03-07T23:39:03.170 DEBUG:teuthology.orchestra.run.smithi166:> sudo chmod 0644 /etc/ceph/ceph.client.b.keyring 2022-03-07T23:39:03.191 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph auth get-or-create client.c mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-03-07T23:39:03.972 INFO:teuthology.orchestra.run.smithi078.stdout:[client.c] 2022-03-07T23:39:03.973 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCXlyZiZDgnMBAA4PkwrShuzka7+yxTGlpyeA== 2022-03-07T23:39:03.995 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2022-03-07T23:39:03.996 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.client.c.keyring 2022-03-07T23:39:03.996 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0644 /etc/ceph/ceph.client.c.keyring 2022-03-07T23:39:04.021 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd dump --format=json 2022-03-07T23:39:04.633 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:39:04.634 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":35,"fsid":"73982954-8071-44d2-bbe9-015898f85301","created":"2022-03-07T23:34:12.293392+0000","modified":"2022-03-07T23:38:58.036812+0000","last_up_change":"2022-03-07T23:38:55.992734+0000","last_in_change":"2022-03-07T23:38:14.810738+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":15,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-07T23:38:39.704918+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"e89e648b-a7c1-42c9-b183-fcdf40cc54c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.12:6800","nonce":476559293},{"type":"v1","addr":"10.250.104.12:6801","nonce":476559293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.12:6802","nonce":476559293},{"type":"v1","addr":"10.250.104.12:6803","nonce":476559293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.12:6806","nonce":476559293},{"type":"v1","addr":"10.250.104.12:6807","nonce":476559293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.12:6804","nonce":476559293},{"type":"v1","addr":"10.250.104.12:6805","nonce":476559293}]},"public_addr":"10.250.104.12:6801/476559293","cluster_addr":"10.250.104.12:6803/476559293","heartbeat_back_addr":"10.250.104.12:6807/476559293","heartbeat_front_addr":"10.250.104.12:6805/476559293","state":["exists","up"]},{"osd":1,"uuid":"1657736a-af22-494d-af8c-c09b43332521","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.16:6800","nonce":505851920},{"type":"v1","addr":"10.250.105.16:6801","nonce":505851920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.16:6802","nonce":505851920},{"type":"v1","addr":"10.250.105.16:6803","nonce":505851920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.16:6806","nonce":505851920},{"type":"v1","addr":"10.250.105.16:6807","nonce":505851920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.16:6804","nonce":505851920},{"type":"v1","addr":"10.250.105.16:6805","nonce":505851920}]},"public_addr":"10.250.105.16:6801/505851920","cluster_addr":"10.250.105.16:6803/505851920","heartbeat_back_addr":"10.250.105.16:6807/505851920","heartbeat_front_addr":"10.250.105.16:6805/505851920","state":["exists","up"]},{"osd":2,"uuid":"a8a285e0-57ee-432b-9f49-2ee77dc4d528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.13:6800","nonce":2511285357},{"type":"v1","addr":"10.250.106.13:6801","nonce":2511285357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.13:6802","nonce":2511285357},{"type":"v1","addr":"10.250.106.13:6803","nonce":2511285357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.13:6806","nonce":2511285357},{"type":"v1","addr":"10.250.106.13:6807","nonce":2511285357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.13:6804","nonce":2511285357},{"type":"v1","addr":"10.250.106.13:6805","nonce":2511285357}]},"public_addr":"10.250.106.13:6801/2511285357","cluster_addr":"10.250.106.13:6803/2511285357","heartbeat_back_addr":"10.250.106.13:6807/2511285357","heartbeat_front_addr":"10.250.106.13:6805/2511285357","state":["exists","up"]},{"osd":3,"uuid":"8f6033d9-aae6-420b-8e13-60f003e917f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.17:6800","nonce":3493190379},{"type":"v1","addr":"10.250.105.17:6801","nonce":3493190379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.17:6802","nonce":3493190379},{"type":"v1","addr":"10.250.105.17:6803","nonce":3493190379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.17:6806","nonce":3493190379},{"type":"v1","addr":"10.250.105.17:6807","nonce":3493190379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.17:6804","nonce":3493190379},{"type":"v1","addr":"10.250.105.17:6805","nonce":3493190379}]},"public_addr":"10.250.105.17:6801/3493190379","cluster_addr":"10.250.105.17:6803/3493190379","heartbeat_back_addr":"10.250.105.17:6807/3493190379","heartbeat_front_addr":"10.250.105.17:6805/3493190379","state":["exists","up"]},{"osd":4,"uuid":"bc55347b-31bc-40d9-a210-e4b9741da05a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.14:6800","nonce":3495326819},{"type":"v1","addr":"10.250.106.14:6801","nonce":3495326819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.14:6802","nonce":3495326819},{"type":"v1","addr":"10.250.106.14:6803","nonce":3495326819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.14:6806","nonce":3495326819},{"type":"v1","addr":"10.250.106.14:6807","nonce":3495326819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.14:6804","nonce":3495326819},{"type":"v1","addr":"10.250.106.14:6805","nonce":3495326819}]},"public_addr":"10.250.106.14:6801/3495326819","cluster_addr":"10.250.106.14:6803/3495326819","heartbeat_back_addr":"10.250.106.14:6807/3495326819","heartbeat_front_addr":"10.250.106.14:6805/3495326819","state":["exists","up"]},{"osd":5,"uuid":"f955637a-c8c4-4821-9ea0-a0a9c565aa5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.18:6800","nonce":3011431620},{"type":"v1","addr":"10.250.105.18:6801","nonce":3011431620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.18:6802","nonce":3011431620},{"type":"v1","addr":"10.250.105.18:6803","nonce":3011431620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.18:6806","nonce":3011431620},{"type":"v1","addr":"10.250.105.18:6807","nonce":3011431620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.18:6804","nonce":3011431620},{"type":"v1","addr":"10.250.105.18:6805","nonce":3011431620}]},"public_addr":"10.250.105.18:6801/3011431620","cluster_addr":"10.250.105.18:6803/3011431620","heartbeat_back_addr":"10.250.105.18:6807/3011431620","heartbeat_front_addr":"10.250.105.18:6805/3011431620","state":["exists","up"]},{"osd":6,"uuid":"12d44d5d-0655-479f-af97-b4cb667d911e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.15:6800","nonce":1274376622},{"type":"v1","addr":"10.250.106.15:6801","nonce":1274376622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.15:6802","nonce":1274376622},{"type":"v1","addr":"10.250.106.15:6803","nonce":1274376622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.15:6806","nonce":1274376622},{"type":"v1","addr":"10.250.106.15:6807","nonce":1274376622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.15:6804","nonce":1274376622},{"type":"v1","addr":"10.250.106.15:6805","nonce":1274376622}]},"public_addr":"10.250.106.15:6801/1274376622","cluster_addr":"10.250.106.15:6803/1274376622","heartbeat_back_addr":"10.250.106.15:6807/1274376622","heartbeat_front_addr":"10.250.106.15:6805/1274376622","state":["exists","up"]},{"osd":7,"uuid":"3c9a93d5-2e95-4544-89b5-b0bf9a29c4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.19:6800","nonce":1090661236},{"type":"v1","addr":"10.250.105.19:6801","nonce":1090661236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.19:6802","nonce":1090661236},{"type":"v1","addr":"10.250.105.19:6803","nonce":1090661236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.19:6806","nonce":1090661236},{"type":"v1","addr":"10.250.105.19:6807","nonce":1090661236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.19:6804","nonce":1090661236},{"type":"v1","addr":"10.250.105.19:6805","nonce":1090661236}]},"public_addr":"10.250.105.19:6801/1090661236","cluster_addr":"10.250.105.19:6803/1090661236","heartbeat_back_addr":"10.250.105.19:6807/1090661236","heartbeat_front_addr":"10.250.105.19:6805/1090661236","state":["exists","up"]},{"osd":8,"uuid":"92c2d248-8274-44f6-849d-69450564261f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.16:6800","nonce":2150246511},{"type":"v1","addr":"10.250.106.16:6801","nonce":2150246511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.16:6802","nonce":2150246511},{"type":"v1","addr":"10.250.106.16:6803","nonce":2150246511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.16:6806","nonce":2150246511},{"type":"v1","addr":"10.250.106.16:6807","nonce":2150246511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.16:6804","nonce":2150246511},{"type":"v1","addr":"10.250.106.16:6805","nonce":2150246511}]},"public_addr":"10.250.106.16:6801/2150246511","cluster_addr":"10.250.106.16:6803/2150246511","heartbeat_back_addr":"10.250.106.16:6807/2150246511","heartbeat_front_addr":"10.250.106.16:6805/2150246511","state":["exists","up"]},{"osd":9,"uuid":"7bf31b02-6f03-446c-834f-7ccf89af7e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.13:6800","nonce":3316042872},{"type":"v1","addr":"10.250.104.13:6801","nonce":3316042872}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.13:6802","nonce":3316042872},{"type":"v1","addr":"10.250.104.13:6803","nonce":3316042872}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.13:6806","nonce":3316042872},{"type":"v1","addr":"10.250.104.13:6807","nonce":3316042872}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.13:6804","nonce":3316042872},{"type":"v1","addr":"10.250.104.13:6805","nonce":3316042872}]},"public_addr":"10.250.104.13:6801/3316042872","cluster_addr":"10.250.104.13:6803/3316042872","heartbeat_back_addr":"10.250.104.13:6807/3316042872","heartbeat_front_addr":"10.250.104.13:6805/3316042872","state":["exists","up"]},{"osd":10,"uuid":"6c7c347a-fe2f-4dc1-a1aa-d02f8fe23014","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.14:6800","nonce":861780288},{"type":"v1","addr":"10.250.104.14:6801","nonce":861780288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.14:6802","nonce":861780288},{"type":"v1","addr":"10.250.104.14:6803","nonce":861780288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.14:6806","nonce":861780288},{"type":"v1","addr":"10.250.104.14:6807","nonce":861780288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.14:6804","nonce":861780288},{"type":"v1","addr":"10.250.104.14:6805","nonce":861780288}]},"public_addr":"10.250.104.14:6801/861780288","cluster_addr":"10.250.104.14:6803/861780288","heartbeat_back_addr":"10.250.104.14:6807/861780288","heartbeat_front_addr":"10.250.104.14:6805/861780288","state":["exists","up"]},{"osd":11,"uuid":"599e0632-73ab-4372-8dad-b201a542f1fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.15:6800","nonce":3151114687},{"type":"v1","addr":"10.250.104.15:6801","nonce":3151114687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.15:6802","nonce":3151114687},{"type":"v1","addr":"10.250.104.15:6803","nonce":3151114687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.15:6806","nonce":3151114687},{"type":"v1","addr":"10.250.104.15:6807","nonce":3151114687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.15:6804","nonce":3151114687},{"type":"v1","addr":"10.250.104.15:6805","nonce":3151114687}]},"public_addr":"10.250.104.15:6801/3151114687","cluster_addr":"10.250.104.15:6803/3151114687","heartbeat_back_addr":"10.250.104.15:6807/3151114687","heartbeat_front_addr":"10.250.104.15:6805/3151114687","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:32.808211+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:28.276504+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:33.278791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:32.438986+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:41.165868+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:41.221619+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:41.425814+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:41.081764+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:44.307592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:50.125406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:54.053128+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:53.976080+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.250.104.6:6800/2676844494":"2022-03-08T23:37:06.817101+0000","10.250.104.6:6801/2676844494":"2022-03-08T23:37:06.817101+0000","10.250.104.6:0/2206850761":"2022-03-08T23:37:06.817101+0000","10.250.104.6:0/2832980667":"2022-03-08T23:37:06.817101+0000","10.250.104.6:0/3040354870":"2022-03-08T23:37:06.817101+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-07T23:39:04.651 INFO:tasks.rook.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-03-07T23:38:39.704918+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'on', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '30', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-03-07T23:39:04.652 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd pool get .mgr pg_num 2022-03-07T23:39:05.173 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2022-03-07T23:39:05.206 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-03-07T23:39:05.207 INFO:tasks.rook.ceph_manager.ceph:waiting for mgr available 2022-03-07T23:39:05.207 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph mgr dump --format=json 2022-03-07T23:39:05.755 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:39:05.776 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":13,"active_gid":24206,"active_name":"a","active_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.6:6800","nonce":3083249627},{"type":"v1","addr":"10.250.104.6:6801","nonce":3083249627}]},"active_addr":"10.250.104.6:6801/3083249627","active_change":"2022-03-07T23:37:06.859640+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","prometheus","restful","rook"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"0.0.0.0","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"prometheus":"http://10.250.104.6:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":2,"active_clients":[{"addrvec":[{"type":"v2","addr":"10.250.104.6:0","nonce":2569750958}]},{"addrvec":[{"type":"v2","addr":"10.250.104.6:0","nonce":2770082691}]},{"addrvec":[{"type":"v2","addr":"10.250.104.6:0","nonce":1385059414}]},{"addrvec":[{"type":"v2","addr":"10.250.104.6:0","nonce":3540393240}]}]}} 2022-03-07T23:39:05.779 INFO:tasks.rook.ceph_manager.ceph:mgr available! 2022-03-07T23:39:05.779 INFO:tasks.rook.ceph_manager.ceph:waiting for all up 2022-03-07T23:39:05.780 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd dump --format=json 2022-03-07T23:39:06.291 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:39:06.291 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":35,"fsid":"73982954-8071-44d2-bbe9-015898f85301","created":"2022-03-07T23:34:12.293392+0000","modified":"2022-03-07T23:38:58.036812+0000","last_up_change":"2022-03-07T23:38:55.992734+0000","last_in_change":"2022-03-07T23:38:14.810738+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":15,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-07T23:38:39.704918+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"e89e648b-a7c1-42c9-b183-fcdf40cc54c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.12:6800","nonce":476559293},{"type":"v1","addr":"10.250.104.12:6801","nonce":476559293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.12:6802","nonce":476559293},{"type":"v1","addr":"10.250.104.12:6803","nonce":476559293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.12:6806","nonce":476559293},{"type":"v1","addr":"10.250.104.12:6807","nonce":476559293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.12:6804","nonce":476559293},{"type":"v1","addr":"10.250.104.12:6805","nonce":476559293}]},"public_addr":"10.250.104.12:6801/476559293","cluster_addr":"10.250.104.12:6803/476559293","heartbeat_back_addr":"10.250.104.12:6807/476559293","heartbeat_front_addr":"10.250.104.12:6805/476559293","state":["exists","up"]},{"osd":1,"uuid":"1657736a-af22-494d-af8c-c09b43332521","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.16:6800","nonce":505851920},{"type":"v1","addr":"10.250.105.16:6801","nonce":505851920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.16:6802","nonce":505851920},{"type":"v1","addr":"10.250.105.16:6803","nonce":505851920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.16:6806","nonce":505851920},{"type":"v1","addr":"10.250.105.16:6807","nonce":505851920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.16:6804","nonce":505851920},{"type":"v1","addr":"10.250.105.16:6805","nonce":505851920}]},"public_addr":"10.250.105.16:6801/505851920","cluster_addr":"10.250.105.16:6803/505851920","heartbeat_back_addr":"10.250.105.16:6807/505851920","heartbeat_front_addr":"10.250.105.16:6805/505851920","state":["exists","up"]},{"osd":2,"uuid":"a8a285e0-57ee-432b-9f49-2ee77dc4d528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.13:6800","nonce":2511285357},{"type":"v1","addr":"10.250.106.13:6801","nonce":2511285357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.13:6802","nonce":2511285357},{"type":"v1","addr":"10.250.106.13:6803","nonce":2511285357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.13:6806","nonce":2511285357},{"type":"v1","addr":"10.250.106.13:6807","nonce":2511285357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.13:6804","nonce":2511285357},{"type":"v1","addr":"10.250.106.13:6805","nonce":2511285357}]},"public_addr":"10.250.106.13:6801/2511285357","cluster_addr":"10.250.106.13:6803/2511285357","heartbeat_back_addr":"10.250.106.13:6807/2511285357","heartbeat_front_addr":"10.250.106.13:6805/2511285357","state":["exists","up"]},{"osd":3,"uuid":"8f6033d9-aae6-420b-8e13-60f003e917f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.17:6800","nonce":3493190379},{"type":"v1","addr":"10.250.105.17:6801","nonce":3493190379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.17:6802","nonce":3493190379},{"type":"v1","addr":"10.250.105.17:6803","nonce":3493190379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.17:6806","nonce":3493190379},{"type":"v1","addr":"10.250.105.17:6807","nonce":3493190379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.17:6804","nonce":3493190379},{"type":"v1","addr":"10.250.105.17:6805","nonce":3493190379}]},"public_addr":"10.250.105.17:6801/3493190379","cluster_addr":"10.250.105.17:6803/3493190379","heartbeat_back_addr":"10.250.105.17:6807/3493190379","heartbeat_front_addr":"10.250.105.17:6805/3493190379","state":["exists","up"]},{"osd":4,"uuid":"bc55347b-31bc-40d9-a210-e4b9741da05a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.14:6800","nonce":3495326819},{"type":"v1","addr":"10.250.106.14:6801","nonce":3495326819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.14:6802","nonce":3495326819},{"type":"v1","addr":"10.250.106.14:6803","nonce":3495326819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.14:6806","nonce":3495326819},{"type":"v1","addr":"10.250.106.14:6807","nonce":3495326819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.14:6804","nonce":3495326819},{"type":"v1","addr":"10.250.106.14:6805","nonce":3495326819}]},"public_addr":"10.250.106.14:6801/3495326819","cluster_addr":"10.250.106.14:6803/3495326819","heartbeat_back_addr":"10.250.106.14:6807/3495326819","heartbeat_front_addr":"10.250.106.14:6805/3495326819","state":["exists","up"]},{"osd":5,"uuid":"f955637a-c8c4-4821-9ea0-a0a9c565aa5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.18:6800","nonce":3011431620},{"type":"v1","addr":"10.250.105.18:6801","nonce":3011431620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.18:6802","nonce":3011431620},{"type":"v1","addr":"10.250.105.18:6803","nonce":3011431620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.18:6806","nonce":3011431620},{"type":"v1","addr":"10.250.105.18:6807","nonce":3011431620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.18:6804","nonce":3011431620},{"type":"v1","addr":"10.250.105.18:6805","nonce":3011431620}]},"public_addr":"10.250.105.18:6801/3011431620","cluster_addr":"10.250.105.18:6803/3011431620","heartbeat_back_addr":"10.250.105.18:6807/3011431620","heartbeat_front_addr":"10.250.105.18:6805/3011431620","state":["exists","up"]},{"osd":6,"uuid":"12d44d5d-0655-479f-af97-b4cb667d911e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.15:6800","nonce":1274376622},{"type":"v1","addr":"10.250.106.15:6801","nonce":1274376622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.15:6802","nonce":1274376622},{"type":"v1","addr":"10.250.106.15:6803","nonce":1274376622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.15:6806","nonce":1274376622},{"type":"v1","addr":"10.250.106.15:6807","nonce":1274376622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.15:6804","nonce":1274376622},{"type":"v1","addr":"10.250.106.15:6805","nonce":1274376622}]},"public_addr":"10.250.106.15:6801/1274376622","cluster_addr":"10.250.106.15:6803/1274376622","heartbeat_back_addr":"10.250.106.15:6807/1274376622","heartbeat_front_addr":"10.250.106.15:6805/1274376622","state":["exists","up"]},{"osd":7,"uuid":"3c9a93d5-2e95-4544-89b5-b0bf9a29c4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.19:6800","nonce":1090661236},{"type":"v1","addr":"10.250.105.19:6801","nonce":1090661236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.19:6802","nonce":1090661236},{"type":"v1","addr":"10.250.105.19:6803","nonce":1090661236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.19:6806","nonce":1090661236},{"type":"v1","addr":"10.250.105.19:6807","nonce":1090661236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.19:6804","nonce":1090661236},{"type":"v1","addr":"10.250.105.19:6805","nonce":1090661236}]},"public_addr":"10.250.105.19:6801/1090661236","cluster_addr":"10.250.105.19:6803/1090661236","heartbeat_back_addr":"10.250.105.19:6807/1090661236","heartbeat_front_addr":"10.250.105.19:6805/1090661236","state":["exists","up"]},{"osd":8,"uuid":"92c2d248-8274-44f6-849d-69450564261f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.16:6800","nonce":2150246511},{"type":"v1","addr":"10.250.106.16:6801","nonce":2150246511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.16:6802","nonce":2150246511},{"type":"v1","addr":"10.250.106.16:6803","nonce":2150246511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.16:6806","nonce":2150246511},{"type":"v1","addr":"10.250.106.16:6807","nonce":2150246511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.16:6804","nonce":2150246511},{"type":"v1","addr":"10.250.106.16:6805","nonce":2150246511}]},"public_addr":"10.250.106.16:6801/2150246511","cluster_addr":"10.250.106.16:6803/2150246511","heartbeat_back_addr":"10.250.106.16:6807/2150246511","heartbeat_front_addr":"10.250.106.16:6805/2150246511","state":["exists","up"]},{"osd":9,"uuid":"7bf31b02-6f03-446c-834f-7ccf89af7e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.13:6800","nonce":3316042872},{"type":"v1","addr":"10.250.104.13:6801","nonce":3316042872}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.13:6802","nonce":3316042872},{"type":"v1","addr":"10.250.104.13:6803","nonce":3316042872}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.13:6806","nonce":3316042872},{"type":"v1","addr":"10.250.104.13:6807","nonce":3316042872}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.13:6804","nonce":3316042872},{"type":"v1","addr":"10.250.104.13:6805","nonce":3316042872}]},"public_addr":"10.250.104.13:6801/3316042872","cluster_addr":"10.250.104.13:6803/3316042872","heartbeat_back_addr":"10.250.104.13:6807/3316042872","heartbeat_front_addr":"10.250.104.13:6805/3316042872","state":["exists","up"]},{"osd":10,"uuid":"6c7c347a-fe2f-4dc1-a1aa-d02f8fe23014","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.14:6800","nonce":861780288},{"type":"v1","addr":"10.250.104.14:6801","nonce":861780288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.14:6802","nonce":861780288},{"type":"v1","addr":"10.250.104.14:6803","nonce":861780288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.14:6806","nonce":861780288},{"type":"v1","addr":"10.250.104.14:6807","nonce":861780288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.14:6804","nonce":861780288},{"type":"v1","addr":"10.250.104.14:6805","nonce":861780288}]},"public_addr":"10.250.104.14:6801/861780288","cluster_addr":"10.250.104.14:6803/861780288","heartbeat_back_addr":"10.250.104.14:6807/861780288","heartbeat_front_addr":"10.250.104.14:6805/861780288","state":["exists","up"]},{"osd":11,"uuid":"599e0632-73ab-4372-8dad-b201a542f1fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.15:6800","nonce":3151114687},{"type":"v1","addr":"10.250.104.15:6801","nonce":3151114687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.15:6802","nonce":3151114687},{"type":"v1","addr":"10.250.104.15:6803","nonce":3151114687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.15:6806","nonce":3151114687},{"type":"v1","addr":"10.250.104.15:6807","nonce":3151114687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.15:6804","nonce":3151114687},{"type":"v1","addr":"10.250.104.15:6805","nonce":3151114687}]},"public_addr":"10.250.104.15:6801/3151114687","cluster_addr":"10.250.104.15:6803/3151114687","heartbeat_back_addr":"10.250.104.15:6807/3151114687","heartbeat_front_addr":"10.250.104.15:6805/3151114687","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:32.808211+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:28.276504+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:33.278791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:32.438986+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:41.165868+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:41.221619+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:41.425814+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:41.081764+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:44.307592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:50.125406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:54.053128+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:53.976080+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.250.104.6:6800/2676844494":"2022-03-08T23:37:06.817101+0000","10.250.104.6:6801/2676844494":"2022-03-08T23:37:06.817101+0000","10.250.104.6:0/2206850761":"2022-03-08T23:37:06.817101+0000","10.250.104.6:0/2832980667":"2022-03-08T23:37:06.817101+0000","10.250.104.6:0/3040354870":"2022-03-08T23:37:06.817101+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-07T23:39:06.310 INFO:tasks.rook.ceph_manager.ceph:all up! 2022-03-07T23:39:06.311 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd dump --format=json 2022-03-07T23:39:06.860 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:39:06.861 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":35,"fsid":"73982954-8071-44d2-bbe9-015898f85301","created":"2022-03-07T23:34:12.293392+0000","modified":"2022-03-07T23:38:58.036812+0000","last_up_change":"2022-03-07T23:38:55.992734+0000","last_in_change":"2022-03-07T23:38:14.810738+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":15,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-03-07T23:38:39.704918+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"on","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"e89e648b-a7c1-42c9-b183-fcdf40cc54c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.12:6800","nonce":476559293},{"type":"v1","addr":"10.250.104.12:6801","nonce":476559293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.12:6802","nonce":476559293},{"type":"v1","addr":"10.250.104.12:6803","nonce":476559293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.12:6806","nonce":476559293},{"type":"v1","addr":"10.250.104.12:6807","nonce":476559293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.12:6804","nonce":476559293},{"type":"v1","addr":"10.250.104.12:6805","nonce":476559293}]},"public_addr":"10.250.104.12:6801/476559293","cluster_addr":"10.250.104.12:6803/476559293","heartbeat_back_addr":"10.250.104.12:6807/476559293","heartbeat_front_addr":"10.250.104.12:6805/476559293","state":["exists","up"]},{"osd":1,"uuid":"1657736a-af22-494d-af8c-c09b43332521","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.16:6800","nonce":505851920},{"type":"v1","addr":"10.250.105.16:6801","nonce":505851920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.16:6802","nonce":505851920},{"type":"v1","addr":"10.250.105.16:6803","nonce":505851920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.16:6806","nonce":505851920},{"type":"v1","addr":"10.250.105.16:6807","nonce":505851920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.16:6804","nonce":505851920},{"type":"v1","addr":"10.250.105.16:6805","nonce":505851920}]},"public_addr":"10.250.105.16:6801/505851920","cluster_addr":"10.250.105.16:6803/505851920","heartbeat_back_addr":"10.250.105.16:6807/505851920","heartbeat_front_addr":"10.250.105.16:6805/505851920","state":["exists","up"]},{"osd":2,"uuid":"a8a285e0-57ee-432b-9f49-2ee77dc4d528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.13:6800","nonce":2511285357},{"type":"v1","addr":"10.250.106.13:6801","nonce":2511285357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.13:6802","nonce":2511285357},{"type":"v1","addr":"10.250.106.13:6803","nonce":2511285357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.13:6806","nonce":2511285357},{"type":"v1","addr":"10.250.106.13:6807","nonce":2511285357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.13:6804","nonce":2511285357},{"type":"v1","addr":"10.250.106.13:6805","nonce":2511285357}]},"public_addr":"10.250.106.13:6801/2511285357","cluster_addr":"10.250.106.13:6803/2511285357","heartbeat_back_addr":"10.250.106.13:6807/2511285357","heartbeat_front_addr":"10.250.106.13:6805/2511285357","state":["exists","up"]},{"osd":3,"uuid":"8f6033d9-aae6-420b-8e13-60f003e917f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.17:6800","nonce":3493190379},{"type":"v1","addr":"10.250.105.17:6801","nonce":3493190379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.17:6802","nonce":3493190379},{"type":"v1","addr":"10.250.105.17:6803","nonce":3493190379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.17:6806","nonce":3493190379},{"type":"v1","addr":"10.250.105.17:6807","nonce":3493190379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.17:6804","nonce":3493190379},{"type":"v1","addr":"10.250.105.17:6805","nonce":3493190379}]},"public_addr":"10.250.105.17:6801/3493190379","cluster_addr":"10.250.105.17:6803/3493190379","heartbeat_back_addr":"10.250.105.17:6807/3493190379","heartbeat_front_addr":"10.250.105.17:6805/3493190379","state":["exists","up"]},{"osd":4,"uuid":"bc55347b-31bc-40d9-a210-e4b9741da05a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.14:6800","nonce":3495326819},{"type":"v1","addr":"10.250.106.14:6801","nonce":3495326819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.14:6802","nonce":3495326819},{"type":"v1","addr":"10.250.106.14:6803","nonce":3495326819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.14:6806","nonce":3495326819},{"type":"v1","addr":"10.250.106.14:6807","nonce":3495326819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.14:6804","nonce":3495326819},{"type":"v1","addr":"10.250.106.14:6805","nonce":3495326819}]},"public_addr":"10.250.106.14:6801/3495326819","cluster_addr":"10.250.106.14:6803/3495326819","heartbeat_back_addr":"10.250.106.14:6807/3495326819","heartbeat_front_addr":"10.250.106.14:6805/3495326819","state":["exists","up"]},{"osd":5,"uuid":"f955637a-c8c4-4821-9ea0-a0a9c565aa5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.18:6800","nonce":3011431620},{"type":"v1","addr":"10.250.105.18:6801","nonce":3011431620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.18:6802","nonce":3011431620},{"type":"v1","addr":"10.250.105.18:6803","nonce":3011431620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.18:6806","nonce":3011431620},{"type":"v1","addr":"10.250.105.18:6807","nonce":3011431620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.18:6804","nonce":3011431620},{"type":"v1","addr":"10.250.105.18:6805","nonce":3011431620}]},"public_addr":"10.250.105.18:6801/3011431620","cluster_addr":"10.250.105.18:6803/3011431620","heartbeat_back_addr":"10.250.105.18:6807/3011431620","heartbeat_front_addr":"10.250.105.18:6805/3011431620","state":["exists","up"]},{"osd":6,"uuid":"12d44d5d-0655-479f-af97-b4cb667d911e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.15:6800","nonce":1274376622},{"type":"v1","addr":"10.250.106.15:6801","nonce":1274376622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.15:6802","nonce":1274376622},{"type":"v1","addr":"10.250.106.15:6803","nonce":1274376622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.15:6806","nonce":1274376622},{"type":"v1","addr":"10.250.106.15:6807","nonce":1274376622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.15:6804","nonce":1274376622},{"type":"v1","addr":"10.250.106.15:6805","nonce":1274376622}]},"public_addr":"10.250.106.15:6801/1274376622","cluster_addr":"10.250.106.15:6803/1274376622","heartbeat_back_addr":"10.250.106.15:6807/1274376622","heartbeat_front_addr":"10.250.106.15:6805/1274376622","state":["exists","up"]},{"osd":7,"uuid":"3c9a93d5-2e95-4544-89b5-b0bf9a29c4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.19:6800","nonce":1090661236},{"type":"v1","addr":"10.250.105.19:6801","nonce":1090661236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.19:6802","nonce":1090661236},{"type":"v1","addr":"10.250.105.19:6803","nonce":1090661236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.19:6806","nonce":1090661236},{"type":"v1","addr":"10.250.105.19:6807","nonce":1090661236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.105.19:6804","nonce":1090661236},{"type":"v1","addr":"10.250.105.19:6805","nonce":1090661236}]},"public_addr":"10.250.105.19:6801/1090661236","cluster_addr":"10.250.105.19:6803/1090661236","heartbeat_back_addr":"10.250.105.19:6807/1090661236","heartbeat_front_addr":"10.250.105.19:6805/1090661236","state":["exists","up"]},{"osd":8,"uuid":"92c2d248-8274-44f6-849d-69450564261f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.16:6800","nonce":2150246511},{"type":"v1","addr":"10.250.106.16:6801","nonce":2150246511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.16:6802","nonce":2150246511},{"type":"v1","addr":"10.250.106.16:6803","nonce":2150246511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.16:6806","nonce":2150246511},{"type":"v1","addr":"10.250.106.16:6807","nonce":2150246511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.106.16:6804","nonce":2150246511},{"type":"v1","addr":"10.250.106.16:6805","nonce":2150246511}]},"public_addr":"10.250.106.16:6801/2150246511","cluster_addr":"10.250.106.16:6803/2150246511","heartbeat_back_addr":"10.250.106.16:6807/2150246511","heartbeat_front_addr":"10.250.106.16:6805/2150246511","state":["exists","up"]},{"osd":9,"uuid":"7bf31b02-6f03-446c-834f-7ccf89af7e70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.13:6800","nonce":3316042872},{"type":"v1","addr":"10.250.104.13:6801","nonce":3316042872}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.13:6802","nonce":3316042872},{"type":"v1","addr":"10.250.104.13:6803","nonce":3316042872}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.13:6806","nonce":3316042872},{"type":"v1","addr":"10.250.104.13:6807","nonce":3316042872}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.13:6804","nonce":3316042872},{"type":"v1","addr":"10.250.104.13:6805","nonce":3316042872}]},"public_addr":"10.250.104.13:6801/3316042872","cluster_addr":"10.250.104.13:6803/3316042872","heartbeat_back_addr":"10.250.104.13:6807/3316042872","heartbeat_front_addr":"10.250.104.13:6805/3316042872","state":["exists","up"]},{"osd":10,"uuid":"6c7c347a-fe2f-4dc1-a1aa-d02f8fe23014","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.14:6800","nonce":861780288},{"type":"v1","addr":"10.250.104.14:6801","nonce":861780288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.14:6802","nonce":861780288},{"type":"v1","addr":"10.250.104.14:6803","nonce":861780288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.14:6806","nonce":861780288},{"type":"v1","addr":"10.250.104.14:6807","nonce":861780288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.14:6804","nonce":861780288},{"type":"v1","addr":"10.250.104.14:6805","nonce":861780288}]},"public_addr":"10.250.104.14:6801/861780288","cluster_addr":"10.250.104.14:6803/861780288","heartbeat_back_addr":"10.250.104.14:6807/861780288","heartbeat_front_addr":"10.250.104.14:6805/861780288","state":["exists","up"]},{"osd":11,"uuid":"599e0632-73ab-4372-8dad-b201a542f1fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.15:6800","nonce":3151114687},{"type":"v1","addr":"10.250.104.15:6801","nonce":3151114687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.15:6802","nonce":3151114687},{"type":"v1","addr":"10.250.104.15:6803","nonce":3151114687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.15:6806","nonce":3151114687},{"type":"v1","addr":"10.250.104.15:6807","nonce":3151114687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.250.104.15:6804","nonce":3151114687},{"type":"v1","addr":"10.250.104.15:6805","nonce":3151114687}]},"public_addr":"10.250.104.15:6801/3151114687","cluster_addr":"10.250.104.15:6803/3151114687","heartbeat_back_addr":"10.250.104.15:6807/3151114687","heartbeat_front_addr":"10.250.104.15:6805/3151114687","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:32.808211+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:28.276504+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:33.278791+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:32.438986+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:41.165868+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:41.221619+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:41.425814+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:41.081764+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:44.307592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:50.125406+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:54.053128+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-03-07T23:38:53.976080+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.250.104.6:6800/2676844494":"2022-03-08T23:37:06.817101+0000","10.250.104.6:6801/2676844494":"2022-03-08T23:37:06.817101+0000","10.250.104.6:0/2206850761":"2022-03-08T23:37:06.817101+0000","10.250.104.6:0/2832980667":"2022-03-08T23:37:06.817101+0000","10.250.104.6:0/3040354870":"2022-03-08T23:37:06.817101+0000"},"erasure_code_profiles":{"default":{"k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-03-07T23:39:06.884 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph tell osd.0 flush_pg_stats 2022-03-07T23:39:06.885 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph tell osd.1 flush_pg_stats 2022-03-07T23:39:06.886 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph tell osd.2 flush_pg_stats 2022-03-07T23:39:06.886 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph tell osd.3 flush_pg_stats 2022-03-07T23:39:06.887 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph tell osd.4 flush_pg_stats 2022-03-07T23:39:06.888 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph tell osd.5 flush_pg_stats 2022-03-07T23:39:06.889 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph tell osd.6 flush_pg_stats 2022-03-07T23:39:06.890 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph tell osd.7 flush_pg_stats 2022-03-07T23:39:06.891 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph tell osd.8 flush_pg_stats 2022-03-07T23:39:06.891 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph tell osd.9 flush_pg_stats 2022-03-07T23:39:06.892 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph tell osd.10 flush_pg_stats 2022-03-07T23:39:06.893 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph tell osd.11 flush_pg_stats 2022-03-07T23:39:07.422 INFO:teuthology.orchestra.run.smithi078.stdout:81604378633 2022-03-07T23:39:07.423 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd last-stat-seq osd.2 2022-03-07T23:39:07.762 INFO:teuthology.orchestra.run.smithi078.stdout:77309411338 2022-03-07T23:39:07.763 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd last-stat-seq osd.1 2022-03-07T23:39:07.766 INFO:teuthology.orchestra.run.smithi078.stdout:107374182407 2022-03-07T23:39:07.766 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd last-stat-seq osd.6 2022-03-07T23:39:07.768 INFO:teuthology.orchestra.run.smithi078.stdout:107374182407 2022-03-07T23:39:07.769 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd last-stat-seq osd.5 2022-03-07T23:39:07.772 INFO:teuthology.orchestra.run.smithi078.stdout:85899345928 2022-03-07T23:39:07.772 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd last-stat-seq osd.3 2022-03-07T23:39:07.774 INFO:teuthology.orchestra.run.smithi078.stdout:133143986182 2022-03-07T23:39:07.774 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd last-stat-seq osd.9 2022-03-07T23:39:07.778 INFO:teuthology.orchestra.run.smithi078.stdout:103079215111 2022-03-07T23:39:07.779 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd last-stat-seq osd.7 2022-03-07T23:39:07.815 INFO:teuthology.orchestra.run.smithi078.stdout:115964116998 2022-03-07T23:39:07.816 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd last-stat-seq osd.8 2022-03-07T23:39:07.989 INFO:teuthology.orchestra.run.smithi078.stdout:111669149705 2022-03-07T23:39:07.990 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd last-stat-seq osd.4 2022-03-07T23:39:08.043 INFO:teuthology.orchestra.run.smithi078.stdout:146028888068 2022-03-07T23:39:08.044 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd last-stat-seq osd.10 2022-03-07T23:39:08.183 INFO:teuthology.orchestra.run.smithi078.stdout:90194313226 2022-03-07T23:39:08.183 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd last-stat-seq osd.0 2022-03-07T23:39:08.213 INFO:teuthology.orchestra.run.smithi078.stdout:81604378632 2022-03-07T23:39:08.256 INFO:teuthology.orchestra.run.smithi078.stdout:146028888069 2022-03-07T23:39:08.257 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd last-stat-seq osd.11 2022-03-07T23:39:08.486 INFO:teuthology.orchestra.run.smithi078.stdout:85899345927 2022-03-07T23:39:08.563 INFO:teuthology.orchestra.run.smithi078.stdout:77309411337 2022-03-07T23:39:08.576 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378633 got 81604378632 for osd.2 2022-03-07T23:39:08.631 INFO:teuthology.orchestra.run.smithi078.stdout:103079215110 2022-03-07T23:39:08.662 INFO:teuthology.orchestra.run.smithi078.stdout:107374182406 2022-03-07T23:39:08.708 INFO:teuthology.orchestra.run.smithi078.stdout:107374182406 2022-03-07T23:39:08.747 INFO:tasks.rook.ceph_manager.ceph:need seq 85899345928 got 85899345927 for osd.3 2022-03-07T23:39:08.752 INFO:tasks.rook.ceph_manager.ceph:need seq 77309411338 got 77309411337 for osd.1 2022-03-07T23:39:08.756 INFO:tasks.rook.ceph_manager.ceph:need seq 103079215111 got 103079215110 for osd.7 2022-03-07T23:39:08.760 INFO:tasks.rook.ceph_manager.ceph:need seq 107374182407 got 107374182406 for osd.6 2022-03-07T23:39:08.764 INFO:tasks.rook.ceph_manager.ceph:need seq 107374182407 got 107374182406 for osd.5 2022-03-07T23:39:08.800 INFO:teuthology.orchestra.run.smithi078.stdout:115964116997 2022-03-07T23:39:08.827 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116998 got 115964116997 for osd.8 2022-03-07T23:39:08.895 INFO:teuthology.orchestra.run.smithi078.stdout:111669149704 2022-03-07T23:39:08.911 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149705 got 111669149704 for osd.4 2022-03-07T23:39:08.928 INFO:teuthology.orchestra.run.smithi078.stdout:133143986181 2022-03-07T23:39:08.945 INFO:tasks.rook.ceph_manager.ceph:need seq 133143986182 got 133143986181 for osd.9 2022-03-07T23:39:09.012 INFO:teuthology.orchestra.run.smithi078.stdout:146028888067 2022-03-07T23:39:09.030 INFO:tasks.rook.ceph_manager.ceph:need seq 146028888068 got 146028888067 for osd.10 2022-03-07T23:39:09.048 INFO:teuthology.orchestra.run.smithi078.stdout:90194313225 2022-03-07T23:39:09.066 INFO:tasks.rook.ceph_manager.ceph:need seq 90194313226 got 90194313225 for osd.0 2022-03-07T23:39:09.076 INFO:teuthology.orchestra.run.smithi078.stdout:146028888068 2022-03-07T23:39:09.093 INFO:tasks.rook.ceph_manager.ceph:need seq 146028888069 got 146028888068 for osd.11 2022-03-07T23:39:09.578 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd last-stat-seq osd.2 2022-03-07T23:39:09.749 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd last-stat-seq osd.3 2022-03-07T23:39:09.752 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd last-stat-seq osd.1 2022-03-07T23:39:09.758 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd last-stat-seq osd.7 2022-03-07T23:39:09.763 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd last-stat-seq osd.6 2022-03-07T23:39:09.766 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd last-stat-seq osd.5 2022-03-07T23:39:09.829 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd last-stat-seq osd.8 2022-03-07T23:39:09.912 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd last-stat-seq osd.4 2022-03-07T23:39:09.946 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd last-stat-seq osd.9 2022-03-07T23:39:10.031 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd last-stat-seq osd.10 2022-03-07T23:39:10.067 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd last-stat-seq osd.0 2022-03-07T23:39:10.095 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph osd last-stat-seq osd.11 2022-03-07T23:39:10.152 INFO:teuthology.orchestra.run.smithi078.stdout:81604378633 2022-03-07T23:39:10.262 INFO:tasks.rook.ceph_manager.ceph:need seq 81604378633 got 81604378633 for osd.2 2022-03-07T23:39:10.263 DEBUG:teuthology.parallel:result is None 2022-03-07T23:39:10.480 INFO:teuthology.orchestra.run.smithi078.stdout:77309411338 2022-03-07T23:39:10.609 INFO:teuthology.orchestra.run.smithi078.stdout:85899345929 2022-03-07T23:39:10.610 INFO:teuthology.orchestra.run.smithi078.stdout:103079215112 2022-03-07T23:39:10.680 INFO:tasks.rook.ceph_manager.ceph:need seq 77309411338 got 77309411338 for osd.1 2022-03-07T23:39:10.681 DEBUG:teuthology.parallel:result is None 2022-03-07T23:39:10.686 INFO:tasks.rook.ceph_manager.ceph:need seq 103079215111 got 103079215112 for osd.7 2022-03-07T23:39:10.686 DEBUG:teuthology.parallel:result is None 2022-03-07T23:39:10.693 INFO:tasks.rook.ceph_manager.ceph:need seq 85899345928 got 85899345929 for osd.3 2022-03-07T23:39:10.694 DEBUG:teuthology.parallel:result is None 2022-03-07T23:39:10.723 INFO:teuthology.orchestra.run.smithi078.stdout:115964116998 2022-03-07T23:39:10.755 INFO:tasks.rook.ceph_manager.ceph:need seq 115964116998 got 115964116998 for osd.8 2022-03-07T23:39:10.755 DEBUG:teuthology.parallel:result is None 2022-03-07T23:39:10.795 INFO:teuthology.orchestra.run.smithi078.stdout:107374182407 2022-03-07T23:39:10.819 INFO:tasks.rook.ceph_manager.ceph:need seq 107374182407 got 107374182407 for osd.5 2022-03-07T23:39:10.820 DEBUG:teuthology.parallel:result is None 2022-03-07T23:39:10.889 INFO:teuthology.orchestra.run.smithi078.stdout:107374182407 2022-03-07T23:39:10.915 INFO:tasks.rook.ceph_manager.ceph:need seq 107374182407 got 107374182407 for osd.6 2022-03-07T23:39:10.915 DEBUG:teuthology.parallel:result is None 2022-03-07T23:39:10.958 INFO:teuthology.orchestra.run.smithi078.stdout:111669149705 2022-03-07T23:39:10.977 INFO:tasks.rook.ceph_manager.ceph:need seq 111669149705 got 111669149705 for osd.4 2022-03-07T23:39:10.977 DEBUG:teuthology.parallel:result is None 2022-03-07T23:39:11.040 INFO:teuthology.orchestra.run.smithi078.stdout:133143986183 2022-03-07T23:39:11.063 INFO:tasks.rook.ceph_manager.ceph:need seq 133143986182 got 133143986183 for osd.9 2022-03-07T23:39:11.064 DEBUG:teuthology.parallel:result is None 2022-03-07T23:39:11.070 INFO:teuthology.orchestra.run.smithi078.stdout:90194313226 2022-03-07T23:39:11.090 INFO:tasks.rook.ceph_manager.ceph:need seq 90194313226 got 90194313226 for osd.0 2022-03-07T23:39:11.091 DEBUG:teuthology.parallel:result is None 2022-03-07T23:39:11.126 INFO:teuthology.orchestra.run.smithi078.stdout:146028888068 2022-03-07T23:39:11.144 INFO:tasks.rook.ceph_manager.ceph:need seq 146028888068 got 146028888068 for osd.10 2022-03-07T23:39:11.145 DEBUG:teuthology.parallel:result is None 2022-03-07T23:39:11.153 INFO:teuthology.orchestra.run.smithi078.stdout:146028888069 2022-03-07T23:39:11.171 INFO:tasks.rook.ceph_manager.ceph:need seq 146028888069 got 146028888069 for osd.11 2022-03-07T23:39:11.171 DEBUG:teuthology.parallel:result is None 2022-03-07T23:39:11.171 INFO:tasks.rook.ceph_manager.ceph:waiting for clean 2022-03-07T23:39:11.172 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph pg dump --format=json 2022-03-07T23:39:11.649 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:39:11.652 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2022-03-07T23:39:11.668 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2022-03-07T23:39:11.104377+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":67084,"kb_used_data":5676,"kb_used_omap":0,"kb_used_meta":60928,"kb_avail":1124874740,"statfs":{"total":1151940427776,"available":1151871733760,"internally_reserved":0,"allocated":5812224,"data_stored":3403558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62390272},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003410"},"pg_stats":[{"pgid":"1.0","version":"35'137","reported_seq":188,"reported_epoch":35,"state":"active+clean","last_fresh":"2022-03-07T23:39:02.203666+0000","last_change":"2022-03-07T23:38:50.273425+0000","last_active":"2022-03-07T23:39:02.203666+0000","last_peered":"2022-03-07T23:39:02.203666+0000","last_clean":"2022-03-07T23:39:02.203666+0000","last_became_active":"2022-03-07T23:38:50.271221+0000","last_became_peered":"2022-03-07T23:38:50.271221+0000","last_unstale":"2022-03-07T23:39:02.203666+0000","last_undegraded":"2022-03-07T23:39:02.203666+0000","last_fullsized":"2022-03-07T23:39:02.203666+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-07T23:38:40.204326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-07T23:38:40.204326+0000","last_clean_scrub_stamp":"2022-03-07T23:38:40.204326+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-09T02:07:45.925090+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":11,"up_from":34,"seq":146028888069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5216,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739936,"statfs":{"total":95995035648,"available":95989694464,"internally_reserved":0,"allocated":352256,"data_stored":151796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.387}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4430000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2509999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.887}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.579}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.016}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6339999999999999}]}]},{"osd":10,"up_from":34,"seq":146028888068,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5216,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739936,"statfs":{"total":95995035648,"available":95989694464,"internally_reserved":0,"allocated":352256,"data_stored":151796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3280000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.365}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.593}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6910000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.744}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2759999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9370000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.919}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3380000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8440000000000001}]}]},{"osd":9,"up_from":31,"seq":133143986183,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5216,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739936,"statfs":{"total":95995035648,"available":95989694464,"internally_reserved":0,"allocated":352256,"data_stored":151796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0369999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.284}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.117}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.986}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.901}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4570000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8029999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.837}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0589999999999999}]}]},{"osd":8,"up_from":27,"seq":115964116998,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6504,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738648,"statfs":{"total":95995035648,"available":95988375552,"internally_reserved":0,"allocated":884736,"data_stored":680724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.296}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.716}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.387}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.095}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8520000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7689999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.881}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.73}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.9809999999999999}]}]},{"osd":4,"up_from":26,"seq":111669149705,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5280,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739872,"statfs":{"total":95995035648,"available":95989628928,"internally_reserved":0,"allocated":352256,"data_stored":151796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7190000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0880000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1469999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6559999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.052}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0739999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0259999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.742}]}]},{"osd":6,"up_from":25,"seq":107374182408,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5280,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739872,"statfs":{"total":95995035648,"available":95989628928,"internally_reserved":0,"allocated":352256,"data_stored":151796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2769999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.867}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.661}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7689999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7969999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7110000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9359999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.635}]}]},{"osd":2,"up_from":19,"seq":81604378633,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6568,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738584,"statfs":{"total":95995035648,"available":95988310016,"internally_reserved":0,"allocated":884736,"data_stored":680724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.087}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5449999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3819999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.347}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6459999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.625}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.681}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7909999999999999}]}]},{"osd":1,"up_from":18,"seq":77309411339,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5340,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739812,"statfs":{"total":95995035648,"available":95989567488,"internally_reserved":0,"allocated":348160,"data_stored":150214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3440000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5549999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9079999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4650000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0790000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1070000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0499999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.125}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.02}]}]},{"osd":3,"up_from":20,"seq":85899345929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5340,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739812,"statfs":{"total":95995035648,"available":95989567488,"internally_reserved":0,"allocated":348160,"data_stored":150182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0429999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4419999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.712}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9650000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.125}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0449999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3440000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.804}]}]},{"osd":0,"up_from":21,"seq":90194313227,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6564,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738588,"statfs":{"total":95995035648,"available":95988314112,"internally_reserved":0,"allocated":880640,"data_stored":679142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.972}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.129}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.26}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3460000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3160000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1970000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.153}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9099999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3919999999999999}]}]},{"osd":7,"up_from":24,"seq":103079215112,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5280,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739872,"statfs":{"total":95995035648,"available":95989628928,"internally_reserved":0,"allocated":352256,"data_stored":151796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.54}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.145}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.665}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.573}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9179999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.696}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.821}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.726}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2160000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9990000000000001}]}]},{"osd":5,"up_from":25,"seq":107374182408,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5280,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739872,"statfs":{"total":95995035648,"available":95989628928,"internally_reserved":0,"allocated":352256,"data_stored":151796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0029999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.508}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5580000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7450000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4590000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.296}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.385}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.363}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.696}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-07T23:39:11.670 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph pg dump --format=json 2022-03-07T23:39:12.224 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:39:12.226 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2022-03-07T23:39:12.243 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2022-03-07T23:39:11.104377+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":67084,"kb_used_data":5676,"kb_used_omap":0,"kb_used_meta":60928,"kb_avail":1124874740,"statfs":{"total":1151940427776,"available":1151871733760,"internally_reserved":0,"allocated":5812224,"data_stored":3403558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62390272},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003410"},"pg_stats":[{"pgid":"1.0","version":"35'137","reported_seq":188,"reported_epoch":35,"state":"active+clean","last_fresh":"2022-03-07T23:39:02.203666+0000","last_change":"2022-03-07T23:38:50.273425+0000","last_active":"2022-03-07T23:39:02.203666+0000","last_peered":"2022-03-07T23:39:02.203666+0000","last_clean":"2022-03-07T23:39:02.203666+0000","last_became_active":"2022-03-07T23:38:50.271221+0000","last_became_peered":"2022-03-07T23:38:50.271221+0000","last_unstale":"2022-03-07T23:39:02.203666+0000","last_undegraded":"2022-03-07T23:39:02.203666+0000","last_fullsized":"2022-03-07T23:39:02.203666+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-03-07T23:38:40.204326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-03-07T23:38:40.204326+0000","last_clean_scrub_stamp":"2022-03-07T23:38:40.204326+0000","objects_scrubbed":0,"log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-03-09T02:07:45.925090+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":11,"up_from":34,"seq":146028888069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5216,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739936,"statfs":{"total":95995035648,"available":95989694464,"internally_reserved":0,"allocated":352256,"data_stored":151796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.387}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4430000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2509999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.887}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.579}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.016}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6339999999999999}]}]},{"osd":10,"up_from":34,"seq":146028888068,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5216,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739936,"statfs":{"total":95995035648,"available":95989694464,"internally_reserved":0,"allocated":352256,"data_stored":151796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3280000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.365}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.593}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6910000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.744}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2759999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9370000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.919}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3380000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8440000000000001}]}]},{"osd":9,"up_from":31,"seq":133143986183,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5216,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739936,"statfs":{"total":95995035648,"available":95989694464,"internally_reserved":0,"allocated":352256,"data_stored":151796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0369999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.284}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.117}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.986}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.901}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4570000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8029999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.837}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0589999999999999}]}]},{"osd":8,"up_from":27,"seq":115964116998,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6504,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738648,"statfs":{"total":95995035648,"available":95988375552,"internally_reserved":0,"allocated":884736,"data_stored":680724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.296}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.716}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.387}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.095}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8520000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7689999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.881}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.73}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.9809999999999999}]}]},{"osd":4,"up_from":26,"seq":111669149705,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5280,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739872,"statfs":{"total":95995035648,"available":95989628928,"internally_reserved":0,"allocated":352256,"data_stored":151796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7190000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0880000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1469999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6559999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.052}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0739999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0259999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.742}]}]},{"osd":6,"up_from":25,"seq":107374182408,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5280,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739872,"statfs":{"total":95995035648,"available":95989628928,"internally_reserved":0,"allocated":352256,"data_stored":151796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2769999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.867}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.661}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7689999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7969999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7110000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9359999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.635}]}]},{"osd":2,"up_from":19,"seq":81604378633,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6568,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738584,"statfs":{"total":95995035648,"available":95988310016,"internally_reserved":0,"allocated":884736,"data_stored":680724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.087}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5449999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3819999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.347}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6459999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.625}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.681}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7909999999999999}]}]},{"osd":1,"up_from":18,"seq":77309411339,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5340,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739812,"statfs":{"total":95995035648,"available":95989567488,"internally_reserved":0,"allocated":348160,"data_stored":150214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3440000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5549999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9079999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4650000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0790000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1070000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0499999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.125}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.02}]}]},{"osd":3,"up_from":20,"seq":85899345929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5340,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739812,"statfs":{"total":95995035648,"available":95989567488,"internally_reserved":0,"allocated":348160,"data_stored":150182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0429999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4419999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.712}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9650000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.125}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0449999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3440000000000001}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.804}]}]},{"osd":0,"up_from":21,"seq":90194313227,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6564,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738588,"statfs":{"total":95995035648,"available":95988314112,"internally_reserved":0,"allocated":880640,"data_stored":679142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.972}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.129}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.26}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3460000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3160000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1970000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.153}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9099999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3919999999999999}]}]},{"osd":7,"up_from":24,"seq":103079215112,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5280,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739872,"statfs":{"total":95995035648,"available":95989628928,"internally_reserved":0,"allocated":352256,"data_stored":151796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.54}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.145}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.665}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.573}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9179999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.696}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.821}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.726}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2160000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9990000000000001}]}]},{"osd":5,"up_from":25,"seq":107374182408,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5280,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739872,"statfs":{"total":95995035648,"available":95989628928,"internally_reserved":0,"allocated":352256,"data_stored":151796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0029999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.508}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5580000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7450000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4590000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.296}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.385}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.363}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.696}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-03-07T23:39:12.245 INFO:tasks.rook.ceph_manager.ceph:clean! 2022-03-07T23:39:12.245 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-03-07T23:39:12.246 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy 2022-03-07T23:39:12.246 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph health --format=json 2022-03-07T23:39:12.810 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:39:12.811 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-03-07T23:39:12.835 INFO:tasks.rook.ceph_manager.ceph:wait_until_healthy done 2022-03-07T23:39:12.835 INFO:tasks.rook:Rook complete, yielding 2022-03-07T23:39:12.836 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-07T23:39:12.848 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph -s 2022-03-07T23:39:13.451 INFO:teuthology.orchestra.run.smithi078.stdout: cluster: 2022-03-07T23:39:13.452 INFO:teuthology.orchestra.run.smithi078.stdout: id: 73982954-8071-44d2-bbe9-015898f85301 2022-03-07T23:39:13.452 INFO:teuthology.orchestra.run.smithi078.stdout: health: HEALTH_OK 2022-03-07T23:39:13.452 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:39:13.452 INFO:teuthology.orchestra.run.smithi078.stdout: services: 2022-03-07T23:39:13.453 INFO:teuthology.orchestra.run.smithi078.stdout: mon: 3 daemons, quorum a,b,c (age 2m) 2022-03-07T23:39:13.453 INFO:teuthology.orchestra.run.smithi078.stdout: mgr: a(active, since 2m) 2022-03-07T23:39:13.453 INFO:teuthology.orchestra.run.smithi078.stdout: osd: 12 osds: 12 up (since 17s), 12 in (since 58s) 2022-03-07T23:39:13.453 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:39:13.454 INFO:teuthology.orchestra.run.smithi078.stdout: data: 2022-03-07T23:39:13.454 INFO:teuthology.orchestra.run.smithi078.stdout: pools: 1 pools, 1 pgs 2022-03-07T23:39:13.454 INFO:teuthology.orchestra.run.smithi078.stdout: objects: 2 objects, 577 KiB 2022-03-07T23:39:13.454 INFO:teuthology.orchestra.run.smithi078.stdout: usage: 66 MiB used, 1.0 TiB / 1.0 TiB avail 2022-03-07T23:39:13.455 INFO:teuthology.orchestra.run.smithi078.stdout: pgs: 1 active+clean 2022-03-07T23:39:13.455 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:39:13.455 INFO:teuthology.orchestra.run.smithi078.stdout: io: 2022-03-07T23:39:13.455 INFO:teuthology.orchestra.run.smithi078.stdout: client: 767 B/s rd, 22 KiB/s wr, 0 op/s rd, 1 op/s wr 2022-03-07T23:39:13.455 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:39:13.478 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch status 2022-03-07T23:39:14.171 INFO:teuthology.orchestra.run.smithi078.stdout:Backend: rook 2022-03-07T23:39:14.172 INFO:teuthology.orchestra.run.smithi078.stdout:Available: Yes 2022-03-07T23:39:14.203 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ps 2022-03-07T23:39:14.751 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-07T23:39:14.751 INFO:teuthology.orchestra.run.smithi078.stdout:crashcollector.crash smithi078 running (79s) 0s ago 79s - - 1c5288aef625 2022-03-07T23:39:14.752 INFO:teuthology.orchestra.run.smithi078.stdout:crashcollector.crash smithi166 running (2m) 0s ago 2m - - 1c5288aef625 2022-03-07T23:39:14.752 INFO:teuthology.orchestra.run.smithi078.stdout:crashcollector.crash smithi170 running (2m) 0s ago 2m - - 1c5288aef625 2022-03-07T23:39:14.752 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.a smithi078 running (2m) 0s ago 2m - - 1c5288aef625 2022-03-07T23:39:14.752 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi170 running (5m) 0s ago 5m - - 1c5288aef625 2022-03-07T23:39:14.753 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi166 running (4m) 0s ago 4m - - 1c5288aef625 2022-03-07T23:39:14.753 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (3m) 0s ago 3m - - 1c5288aef625 2022-03-07T23:39:14.753 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (79s) 0s ago 79s - - 1c5288aef625 2022-03-07T23:39:14.753 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi166 running (75s) 0s ago 75s - - 1c5288aef625 2022-03-07T23:39:14.754 INFO:teuthology.orchestra.run.smithi078.stdout:osd.10 smithi078 running (52s) 0s ago 52s - - 1c5288aef625 2022-03-07T23:39:14.754 INFO:teuthology.orchestra.run.smithi078.stdout:osd.11 smithi078 running (51s) 0s ago 51s - - 1c5288aef625 2022-03-07T23:39:14.754 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi170 running (72s) 0s ago 72s - - 1c5288aef625 2022-03-07T23:39:14.754 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi166 running (69s) 0s ago 70s - - 1c5288aef625 2022-03-07T23:39:14.755 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi170 running (67s) 0s ago 67s - - 1c5288aef625 2022-03-07T23:39:14.755 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi166 running (62s) 0s ago 62s - - 1c5288aef625 2022-03-07T23:39:14.755 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi170 running (62s) 0s ago 62s - - 1c5288aef625 2022-03-07T23:39:14.755 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi166 running (60s) 0s ago 60s - - 1c5288aef625 2022-03-07T23:39:14.755 INFO:teuthology.orchestra.run.smithi078.stdout:osd.8 smithi170 running (58s) 0s ago 58s - - 1c5288aef625 2022-03-07T23:39:14.756 INFO:teuthology.orchestra.run.smithi078.stdout:osd.9 smithi078 running (55s) 0s ago 56s - - 1c5288aef625 2022-03-07T23:39:14.774 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls 2022-03-07T23:39:15.490 INFO:teuthology.orchestra.run.smithi078.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-07T23:39:15.490 INFO:teuthology.orchestra.run.smithi078.stdout:crash 3/3 0s ago 2m * 2022-03-07T23:39:15.491 INFO:teuthology.orchestra.run.smithi078.stdout:mgr 1/1 0s ago 2m count:1 2022-03-07T23:39:15.491 INFO:teuthology.orchestra.run.smithi078.stdout:mon 3/3 0s ago 5m count:3 2022-03-07T23:39:15.491 INFO:teuthology.orchestra.run.smithi078.stdout:osd 24 0s ago 80s 2022-03-07T23:39:15.491 INFO:teuthology.orchestra.run.smithi078.stdout:osd.all-available-devices 0 0s ago - * 2022-03-07T23:39:15.516 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch device ls 2022-03-07T23:39:16.045 INFO:teuthology.orchestra.run.smithi078.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-07T23:39:16.045 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-07T23:39:16.046 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-07T23:39:16.046 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-07T23:39:16.046 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-07T23:39:16.046 INFO:teuthology.orchestra.run.smithi078.stdout:smithi166 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-07T23:39:16.047 INFO:teuthology.orchestra.run.smithi078.stdout:smithi166 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-07T23:39:16.047 INFO:teuthology.orchestra.run.smithi078.stdout:smithi166 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-07T23:39:16.047 INFO:teuthology.orchestra.run.smithi078.stdout:smithi166 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-07T23:39:16.047 INFO:teuthology.orchestra.run.smithi078.stdout:smithi170 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-07T23:39:16.048 INFO:teuthology.orchestra.run.smithi078.stdout:smithi170 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-07T23:39:16.048 INFO:teuthology.orchestra.run.smithi078.stdout:smithi170 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-07T23:39:16.048 INFO:teuthology.orchestra.run.smithi078.stdout:smithi170 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-07T23:39:16.069 INFO:teuthology.run_tasks:Running task exec... 2022-03-07T23:39:16.082 INFO:teuthology.task.exec:Executing custom commands... 2022-03-07T23:39:16.083 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi078.front.sepia.ceph.com 2022-03-07T23:39:16.083 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'set -ex 2022-03-07T23:39:16.084 DEBUG:teuthology.orchestra.run.smithi078:> toolbox() { 2022-03-07T23:39:16.084 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- "$@" 2022-03-07T23:39:16.084 DEBUG:teuthology.orchestra.run.smithi078:> } 2022-03-07T23:39:16.085 DEBUG:teuthology.orchestra.run.smithi078:> orig_num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-07T23:39:16.085 DEBUG:teuthology.orchestra.run.smithi078:> toolbox ceph orch osd rm 0 --force 2022-03-07T23:39:16.085 DEBUG:teuthology.orchestra.run.smithi078:> removed_pv="" 2022-03-07T23:39:16.085 DEBUG:teuthology.orchestra.run.smithi078:> while [ "$removed_pv" = "" ] 2022-03-07T23:39:16.086 DEBUG:teuthology.orchestra.run.smithi078:> do 2022-03-07T23:39:16.086 DEBUG:teuthology.orchestra.run.smithi078:> removed_pv=`kubectl get pv | grep Released | cut -f1 -d " "` 2022-03-07T23:39:16.086 DEBUG:teuthology.orchestra.run.smithi078:> sleep 3s 2022-03-07T23:39:16.087 DEBUG:teuthology.orchestra.run.smithi078:> done 2022-03-07T23:39:16.087 DEBUG:teuthology.orchestra.run.smithi078:> target_path=`kubectl get pv $removed_pv -o jsonpath='"'"'{.spec.local.path}'"'"'` 2022-03-07T23:39:16.087 DEBUG:teuthology.orchestra.run.smithi078:> host=`echo $removed_pv | cut -f1 -d "-"` 2022-03-07T23:39:16.087 DEBUG:teuthology.orchestra.run.smithi078:> toolbox ceph orch device zap $host $target_path --force 2022-03-07T23:39:16.088 DEBUG:teuthology.orchestra.run.smithi078:> zap_completion="0" 2022-03-07T23:39:16.088 DEBUG:teuthology.orchestra.run.smithi078:> while [ "$zap_completion" = "0" ] 2022-03-07T23:39:16.088 DEBUG:teuthology.orchestra.run.smithi078:> do 2022-03-07T23:39:16.089 DEBUG:teuthology.orchestra.run.smithi078:> zap_completion=`kubectl get job -n rook-ceph rook-ceph-device-zap -o jsonpath='"'"'{.status.succeeded.path}'"'"'` 2022-03-07T23:39:16.089 DEBUG:teuthology.orchestra.run.smithi078:> sleep 3s 2022-03-07T23:39:16.089 DEBUG:teuthology.orchestra.run.smithi078:> done 2022-03-07T23:39:16.089 DEBUG:teuthology.orchestra.run.smithi078:> kubectl patch pv $removed_pv -p '"'"'{"spec":{"claimRef": null}}'"'"' 2022-03-07T23:39:16.090 DEBUG:teuthology.orchestra.run.smithi078:> toolbox ceph orch apply osd --all-available-devices 2022-03-07T23:39:16.090 DEBUG:teuthology.orchestra.run.smithi078:> kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-07T23:39:16.090 DEBUG:teuthology.orchestra.run.smithi078:> num_osd="0" 2022-03-07T23:39:16.091 DEBUG:teuthology.orchestra.run.smithi078:> while [ "$num_osd" != "$orig_num_osd" ] 2022-03-07T23:39:16.091 DEBUG:teuthology.orchestra.run.smithi078:> do 2022-03-07T23:39:16.091 DEBUG:teuthology.orchestra.run.smithi078:> echo "waiting for osd to come back up" 2022-03-07T23:39:16.091 DEBUG:teuthology.orchestra.run.smithi078:> num_osd=`toolbox ceph osd stat | cut -f3 -d " "` 2022-03-07T23:39:16.092 DEBUG:teuthology.orchestra.run.smithi078:> sleep 30s 2022-03-07T23:39:16.092 DEBUG:teuthology.orchestra.run.smithi078:> done 2022-03-07T23:39:16.092 DEBUG:teuthology.orchestra.run.smithi078:> ' 2022-03-07T23:39:16.110 INFO:teuthology.orchestra.run.smithi078.stderr:++ toolbox ceph osd stat 2022-03-07T23:39:16.110 INFO:teuthology.orchestra.run.smithi078.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-07T23:39:16.111 INFO:teuthology.orchestra.run.smithi078.stderr:++ cut -f3 -d ' ' 2022-03-07T23:39:18.115 INFO:teuthology.orchestra.run.smithi078.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-07T23:39:18.602 INFO:teuthology.orchestra.run.smithi078.stderr:+ orig_num_osd=12 2022-03-07T23:39:18.602 INFO:teuthology.orchestra.run.smithi078.stderr:+ toolbox ceph orch osd rm 0 --force 2022-03-07T23:39:18.602 INFO:teuthology.orchestra.run.smithi078.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch osd rm 0 --force 2022-03-07T23:39:18.668 INFO:teuthology.orchestra.run.smithi078.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-07T23:39:19.292 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.293242 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-07T23:39:19.292 INFO:tasks.rook.operator.smithi078.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", AllowUnsupported: true}, 2022-03-07T23:39:19.292 INFO:tasks.rook.operator.smithi078.stdout: Storage: v1.StorageScopeSpec{ 2022-03-07T23:39:19.292 INFO:tasks.rook.operator.smithi078.stdout: ... // 3 identical fields 2022-03-07T23:39:19.293 INFO:tasks.rook.operator.smithi078.stdout: Config: nil, 2022-03-07T23:39:19.293 INFO:tasks.rook.operator.smithi078.stdout: Selection: {}, 2022-03-07T23:39:19.294 INFO:tasks.rook.operator.smithi078.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-07T23:39:19.294 INFO:tasks.rook.operator.smithi078.stdout: {Name: "smithi078-lv-1", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi078-lv-1", StorageClassName: &"scratch", ...}}}}, 2022-03-07T23:39:19.296 INFO:tasks.rook.operator.smithi078.stdout:- { 2022-03-07T23:39:19.297 INFO:tasks.rook.operator.smithi078.stdout:- Name: "smithi078-lv-2", 2022-03-07T23:39:19.297 INFO:tasks.rook.operator.smithi078.stdout:- Count: 1, 2022-03-07T23:39:19.297 INFO:tasks.rook.operator.smithi078.stdout:- VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-07T23:39:19.298 INFO:tasks.rook.operator.smithi078.stdout:- }, 2022-03-07T23:39:19.298 INFO:tasks.rook.operator.smithi078.stdout: {Name: "smithi078-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi078-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-07T23:39:19.298 INFO:tasks.rook.operator.smithi078.stdout: {Name: "smithi078-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi078-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-07T23:39:19.298 INFO:tasks.rook.operator.smithi078.stdout: ... // 8 identical elements 2022-03-07T23:39:19.299 INFO:tasks.rook.operator.smithi078.stdout: }, 2022-03-07T23:39:19.299 INFO:tasks.rook.operator.smithi078.stdout: }, 2022-03-07T23:39:19.299 INFO:tasks.rook.operator.smithi078.stdout: Annotations: nil, 2022-03-07T23:39:19.299 INFO:tasks.rook.operator.smithi078.stdout: Labels: nil, 2022-03-07T23:39:19.300 INFO:tasks.rook.operator.smithi078.stdout: ... // 20 identical fields 2022-03-07T23:39:19.300 INFO:tasks.rook.operator.smithi078.stdout: } 2022-03-07T23:39:19.300 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.293425 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-07T23:39:19.300 INFO:tasks.rook.operator.smithi078.stdout:I0307 23:39:19.293642 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-07T23:39:19.301 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.293937 I | operator: successfully started the controller-runtime manager 2022-03-07T23:39:19.303 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.305875 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-07T23:39:19.304 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.305892 I | operator: watching all namespaces for Ceph CRs 2022-03-07T23:39:19.304 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.305919 I | operator: setting up schemes 2022-03-07T23:39:19.307 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.308554 I | operator: setting up the controller-runtime manager 2022-03-07T23:39:19.510 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.512222 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-07T23:39:19.628 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.629250 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-07T23:39:19.628 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.629296 I | ceph-cluster-controller: successfully started 2022-03-07T23:39:19.628 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.629389 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-07T23:39:19.629 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.629431 I | ceph-crashcollector-controller: successfully started 2022-03-07T23:39:19.629 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.629477 I | ceph-block-pool-controller: successfully started 2022-03-07T23:39:19.629 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.629500 I | ceph-object-store-user-controller: successfully started 2022-03-07T23:39:19.629 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.629519 I | ceph-object-realm-controller: successfully started 2022-03-07T23:39:19.630 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.629527 I | ceph-object-zonegroup-controller: successfully started 2022-03-07T23:39:19.630 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.629534 I | ceph-object-zone-controller: successfully started 2022-03-07T23:39:19.630 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.629639 I | ceph-object-controller: successfully started 2022-03-07T23:39:19.630 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.629663 I | ceph-file-controller: successfully started 2022-03-07T23:39:19.631 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.629680 I | ceph-nfs-controller: successfully started 2022-03-07T23:39:19.631 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.629703 I | ceph-rbd-mirror-controller: successfully started 2022-03-07T23:39:19.631 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.629732 I | ceph-client-controller: successfully started 2022-03-07T23:39:19.632 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.629748 I | ceph-filesystem-mirror-controller: successfully started 2022-03-07T23:39:19.632 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.629768 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-07T23:39:19.632 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.629788 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-07T23:39:19.633 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.629804 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-07T23:39:19.633 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.629821 I | ceph-bucket-topic: successfully started 2022-03-07T23:39:19.633 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.629837 I | ceph-bucket-notification: successfully started 2022-03-07T23:39:19.633 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.629846 I | ceph-bucket-notification: successfully started 2022-03-07T23:39:19.634 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.629853 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-07T23:39:19.634 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.631181 I | operator: starting the controller-runtime manager 2022-03-07T23:39:19.733 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.735171 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-07T23:39:19.734 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.735234 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-07T23:39:19.734 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.735250 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-07T23:39:19.734 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.735266 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-07T23:39:19.996 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:19.995307 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-07T23:39:20.089 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:20.090578 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:39:20.090 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:20.090682 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-07T23:39:20.090 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:20.090701 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-07T23:39:20.094 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:20.092841 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:39:20.094 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:20.093060 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-07T23:39:20.094 INFO:tasks.rook.operator.smithi078.stdout:I0307 23:39:20.093160 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-07T23:39:20.237 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:20.238903 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-07T23:39:20.258 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:20.260747 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-07T23:39:20.309 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:20.309757 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-07T23:39:20.309 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:20.309795 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-07T23:39:20.309 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:20.309809 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-07T23:39:20.310 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:20.309820 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-07T23:39:20.311 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:20.309832 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-07T23:39:20.311 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:20.309844 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-07T23:39:20.312 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:20.309861 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-07T23:39:20.312 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:20.309874 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-07T23:39:20.312 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:20.309886 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-07T23:39:20.313 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:20.309901 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-07T23:39:20.313 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:20.309912 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-07T23:39:20.313 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:20.309928 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-07T23:39:20.313 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:20.309941 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-07T23:39:20.314 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:20.309952 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-07T23:39:20.314 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:20.310076 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-07T23:39:20.314 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:20.310094 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-07T23:39:20.314 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:20.310189 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-07T23:39:20.315 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:20.310211 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-07T23:39:23.149 INFO:teuthology.orchestra.run.smithi078.stdout:Success 2022-03-07T23:39:23.171 INFO:teuthology.orchestra.run.smithi078.stderr:+ removed_pv= 2022-03-07T23:39:23.171 INFO:teuthology.orchestra.run.smithi078.stderr:+ '[' '' = '' ']' 2022-03-07T23:39:23.172 INFO:teuthology.orchestra.run.smithi078.stderr:++ grep Released 2022-03-07T23:39:23.172 INFO:teuthology.orchestra.run.smithi078.stderr:++ kubectl get pv 2022-03-07T23:39:23.173 INFO:teuthology.orchestra.run.smithi078.stderr:++ cut -f1 -d ' ' 2022-03-07T23:39:23.284 INFO:teuthology.orchestra.run.smithi078.stderr:+ removed_pv= 2022-03-07T23:39:23.284 INFO:teuthology.orchestra.run.smithi078.stderr:+ sleep 3s 2022-03-07T23:39:23.447 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:23.446778 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-03-07T23:39:26.286 INFO:teuthology.orchestra.run.smithi078.stderr:+ '[' '' = '' ']' 2022-03-07T23:39:26.287 INFO:teuthology.orchestra.run.smithi078.stderr:++ kubectl get pv 2022-03-07T23:39:26.288 INFO:teuthology.orchestra.run.smithi078.stderr:++ grep Released 2022-03-07T23:39:26.288 INFO:teuthology.orchestra.run.smithi078.stderr:++ cut -f1 -d ' ' 2022-03-07T23:39:26.345 INFO:teuthology.orchestra.run.smithi078.stderr:+ removed_pv=smithi078-lv-2 2022-03-07T23:39:26.345 INFO:teuthology.orchestra.run.smithi078.stderr:+ sleep 3s 2022-03-07T23:39:28.044 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.043767 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-07T23:39:28.045 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.043824 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-07T23:39:28.045 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.043843 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-07T23:39:28.045 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.043859 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-07T23:39:28.046 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.043871 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-07T23:39:28.046 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.043887 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-07T23:39:28.046 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.043904 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-07T23:39:28.046 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.043919 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-07T23:39:28.046 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.043936 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-07T23:39:28.047 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.043943 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-07T23:39:28.047 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.043951 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-07T23:39:28.047 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.043963 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-07T23:39:28.047 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.043979 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-07T23:39:28.048 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.043994 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-07T23:39:28.048 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.044002 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-07T23:39:28.048 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.044009 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-07T23:39:28.048 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.044018 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-07T23:39:28.048 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.044026 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-07T23:39:28.049 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.044042 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-07T23:39:28.049 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.044060 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-07T23:39:28.049 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.044077 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-07T23:39:28.049 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.044089 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-07T23:39:28.049 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.044102 I | ceph-csi: Kubernetes version is 1.23 2022-03-07T23:39:28.050 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.044115 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-07T23:39:28.050 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.044132 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-07T23:39:28.077 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.078320 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-07T23:39:28.085 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.084942 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-07T23:39:28.085 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.084958 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-07T23:39:28.086 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.084963 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-07T23:39:28.086 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.084967 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-07T23:39:28.086 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.084971 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-07T23:39:28.086 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.084975 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-07T23:39:28.086 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.084979 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-07T23:39:28.209 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.210362 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-07T23:39:28.210 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.210395 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-07T23:39:28.210 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.210404 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-07T23:39:28.249 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.250812 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-07T23:39:28.422 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.423351 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-07T23:39:28.422 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.423382 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-07T23:39:28.423 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.423397 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-07T23:39:28.513 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.514002 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-07T23:39:28.514 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.514021 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-07T23:39:28.514 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.514025 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-07T23:39:28.557 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.558824 I | ceph-csi: successfully started CSI CephFS driver 2022-03-07T23:39:28.756 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.757071 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-07T23:39:28.761 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.762490 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-07T23:39:28.761 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.762507 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-07T23:39:28.777 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:28.778778 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-07T23:39:29.347 INFO:teuthology.orchestra.run.smithi078.stderr:+ '[' smithi078-lv-2 = '' ']' 2022-03-07T23:39:29.348 INFO:teuthology.orchestra.run.smithi078.stderr:++ kubectl get pv smithi078-lv-2 -o 'jsonpath={.spec.local.path}' 2022-03-07T23:39:29.394 INFO:teuthology.orchestra.run.smithi078.stderr:+ target_path=/dev/vg_nvme/lv_2 2022-03-07T23:39:29.395 INFO:teuthology.orchestra.run.smithi078.stderr:++ cut -f1 -d - 2022-03-07T23:39:29.396 INFO:teuthology.orchestra.run.smithi078.stderr:++ echo smithi078-lv-2 2022-03-07T23:39:29.396 INFO:teuthology.orchestra.run.smithi078.stderr:+ host=smithi078 2022-03-07T23:39:29.397 INFO:teuthology.orchestra.run.smithi078.stderr:+ toolbox ceph orch device zap smithi078 /dev/vg_nvme/lv_2 --force 2022-03-07T23:39:29.397 INFO:teuthology.orchestra.run.smithi078.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch device zap smithi078 /dev/vg_nvme/lv_2 --force 2022-03-07T23:39:29.456 INFO:teuthology.orchestra.run.smithi078.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-07T23:39:29.828 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:29.828864 I | ceph-spec: detected ceph image version: "17.0.0-11002 quincy" 2022-03-07T23:39:29.828 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:29.828895 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-07T23:39:29.828 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:29.828908 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11002 quincy", pursuing 2022-03-07T23:39:29.852 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:29.853736 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:39:29.904 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:29.905013 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T23:39:30.009 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:30.010053 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T23:39:30.295 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/vg_nvme/lv_2 on smithi078 zapped 2022-03-07T23:39:30.316 INFO:teuthology.orchestra.run.smithi078.stderr:+ zap_completion=0 2022-03-07T23:39:30.316 INFO:teuthology.orchestra.run.smithi078.stderr:+ '[' 0 = 0 ']' 2022-03-07T23:39:30.317 INFO:teuthology.orchestra.run.smithi078.stderr:++ kubectl get job -n rook-ceph rook-ceph-device-zap -o 'jsonpath={.status.succeeded.path}' 2022-03-07T23:39:30.421 INFO:teuthology.orchestra.run.smithi078.stderr:+ zap_completion= 2022-03-07T23:39:30.422 INFO:teuthology.orchestra.run.smithi078.stderr:+ sleep 3s 2022-03-07T23:39:30.849 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:30.850333 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11002 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4" 2022-03-07T23:39:31.429 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:31.430741 I | op-mon: start running mons 2022-03-07T23:39:31.606 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:31.607125 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:39:31.835 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:31.835450 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.105.146.196:6789","10.98.222.16:6789","10.96.111.213:6789"]}] data:a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 mapping:{"node":{"a":{"Name":"smithi170","Hostname":"smithi170","Address":"172.21.15.170"},"b":{"Name":"smithi166","Hostname":"smithi166","Address":"172.21.15.166"},"c":{"Name":"smithi078","Hostname":"smithi078","Address":"172.21.15.78"}}} maxMonId:2] 2022-03-07T23:39:31.847 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:31.843584 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T23:39:31.848 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:31.843783 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T23:39:33.220 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:33.221784 I | op-mon: targeting the mon count 3 2022-03-07T23:39:33.227 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:33.228407 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T23:39:33.423 INFO:teuthology.orchestra.run.smithi078.stderr:+ '[' '' = 0 ']' 2022-03-07T23:39:33.423 INFO:teuthology.orchestra.run.smithi078.stderr:+ kubectl patch pv smithi078-lv-2 -p '{"spec":{"claimRef": null}}' 2022-03-07T23:39:33.610 INFO:teuthology.orchestra.run.smithi078.stdout:persistentvolume/smithi078-lv-2 patched 2022-03-07T23:39:33.618 INFO:teuthology.orchestra.run.smithi078.stderr:+ toolbox ceph orch apply osd --all-available-devices 2022-03-07T23:39:33.618 INFO:teuthology.orchestra.run.smithi078.stderr:+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph orch apply osd --all-available-devices 2022-03-07T23:39:33.671 INFO:teuthology.orchestra.run.smithi078.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-07T23:39:33.735 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:33.736370 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T23:39:33.736 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:33.736403 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T23:39:34.101 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.102588 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T23:39:34.102 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.102635 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T23:39:34.269 INFO:teuthology.orchestra.run.smithi078.stdout:['Success'] 2022-03-07T23:39:34.278 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.277179 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff= v1.ClusterSpec{ 2022-03-07T23:39:34.279 INFO:tasks.rook.operator.smithi078.stdout: CephVersion: {Image: "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", AllowUnsupported: true}, 2022-03-07T23:39:34.279 INFO:tasks.rook.operator.smithi078.stdout: Storage: v1.StorageScopeSpec{ 2022-03-07T23:39:34.279 INFO:tasks.rook.operator.smithi078.stdout: ... // 3 identical fields 2022-03-07T23:39:34.279 INFO:tasks.rook.operator.smithi078.stdout: Config: nil, 2022-03-07T23:39:34.280 INFO:tasks.rook.operator.smithi078.stdout: Selection: {}, 2022-03-07T23:39:34.280 INFO:tasks.rook.operator.smithi078.stdout: StorageClassDeviceSets: []v1.StorageClassDeviceSet{ 2022-03-07T23:39:34.280 INFO:tasks.rook.operator.smithi078.stdout: ... // 9 identical elements 2022-03-07T23:39:34.280 INFO:tasks.rook.operator.smithi078.stdout: {Name: "smithi170-lv-3", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi170-lv-3", StorageClassName: &"scratch", ...}}}}, 2022-03-07T23:39:34.281 INFO:tasks.rook.operator.smithi078.stdout: {Name: "smithi170-lv-4", Count: 1, VolumeClaimTemplates: {{ObjectMeta: {Name: "data"}, Spec: {AccessModes: {"ReadWriteOnce"}, Resources: {Requests: {s"storage": {i: {...}, s: "1", Format: "DecimalSI"}}}, VolumeName: "smithi170-lv-4", StorageClassName: &"scratch", ...}}}}, 2022-03-07T23:39:34.281 INFO:tasks.rook.operator.smithi078.stdout:+ { 2022-03-07T23:39:34.281 INFO:tasks.rook.operator.smithi078.stdout:+ Name: "smithi078-lv-2", 2022-03-07T23:39:34.281 INFO:tasks.rook.operator.smithi078.stdout:+ Count: 1, 2022-03-07T23:39:34.282 INFO:tasks.rook.operator.smithi078.stdout:+ VolumeClaimTemplates: []v1.PersistentVolumeClaim{{ObjectMeta: v1.ObjectMeta{...}, Spec: v1.PersistentVolumeClaimSpec{...}}}, 2022-03-07T23:39:34.282 INFO:tasks.rook.operator.smithi078.stdout:+ }, 2022-03-07T23:39:34.282 INFO:tasks.rook.operator.smithi078.stdout: }, 2022-03-07T23:39:34.283 INFO:tasks.rook.operator.smithi078.stdout: }, 2022-03-07T23:39:34.283 INFO:tasks.rook.operator.smithi078.stdout: Annotations: nil, 2022-03-07T23:39:34.283 INFO:tasks.rook.operator.smithi078.stdout: Labels: nil, 2022-03-07T23:39:34.283 INFO:tasks.rook.operator.smithi078.stdout: ... // 20 identical fields 2022-03-07T23:39:34.284 INFO:tasks.rook.operator.smithi078.stdout: } 2022-03-07T23:39:34.284 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.277278 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-07T23:39:34.284 INFO:tasks.rook.operator.smithi078.stdout:I0307 23:39:34.277647 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-07T23:39:34.293 INFO:teuthology.orchestra.run.smithi078.stderr:+ kubectl delete job rook-ceph-device-zap -n rook-ceph 2022-03-07T23:39:34.463 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.463939 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T23:39:34.463 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.463972 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-07T23:39:34.463 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.464046 W | op-mon: failed to set Rook and/or user-defined Ceph config options before starting mons; will retry after starting mons. failed to apply logging configuration: failed to set one or more Ceph configs: failed to set ceph config in the centralized mon configuration database; you may need to use the rook-config-override ConfigMap. output: : context canceled 2022-03-07T23:39:34.464 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.464061 I | op-mon: checking for basic quorum with existing mons 2022-03-07T23:39:34.611 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.612007 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-07T23:39:34.611 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.612035 I | operator: watching all namespaces for Ceph CRs 2022-03-07T23:39:34.612 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.612083 I | operator: setting up schemes 2022-03-07T23:39:34.616 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.613535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph monitors: failed to init mon services: context canceled 2022-03-07T23:39:34.616 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.613680 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-07T23:39:34.617 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.616290 I | operator: setting up the controller-runtime manager 2022-03-07T23:39:34.797 INFO:teuthology.orchestra.run.smithi078.stdout:job.batch "rook-ceph-device-zap" deleted 2022-03-07T23:39:34.823 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.821549 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-07T23:39:34.830 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.829402 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-07T23:39:34.830 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.829435 I | ceph-cluster-controller: successfully started 2022-03-07T23:39:34.831 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.829472 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-07T23:39:34.831 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.829489 I | ceph-crashcollector-controller: successfully started 2022-03-07T23:39:34.832 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.829506 I | ceph-block-pool-controller: successfully started 2022-03-07T23:39:34.832 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.829531 I | ceph-object-store-user-controller: successfully started 2022-03-07T23:39:34.833 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.829554 I | ceph-object-realm-controller: successfully started 2022-03-07T23:39:34.833 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.829565 I | ceph-object-zonegroup-controller: successfully started 2022-03-07T23:39:34.834 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.829583 I | ceph-object-zone-controller: successfully started 2022-03-07T23:39:34.834 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.829712 I | ceph-object-controller: successfully started 2022-03-07T23:39:34.835 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.829758 I | ceph-file-controller: successfully started 2022-03-07T23:39:34.835 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.829807 I | ceph-nfs-controller: successfully started 2022-03-07T23:39:34.837 INFO:teuthology.orchestra.run.smithi078.stderr:+ num_osd=0 2022-03-07T23:39:34.837 INFO:teuthology.orchestra.run.smithi078.stderr:+ '[' 0 '!=' 12 ']' 2022-03-07T23:39:34.838 INFO:teuthology.orchestra.run.smithi078.stderr:+ echo 'waiting for osd to come back up' 2022-03-07T23:39:34.838 INFO:teuthology.orchestra.run.smithi078.stdout:waiting for osd to come back up 2022-03-07T23:39:34.840 INFO:teuthology.orchestra.run.smithi078.stderr:++ toolbox ceph osd stat 2022-03-07T23:39:34.841 INFO:teuthology.orchestra.run.smithi078.stderr:++ cut -f3 -d ' ' 2022-03-07T23:39:34.841 INFO:teuthology.orchestra.run.smithi078.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-07T23:39:34.842 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.829840 I | ceph-rbd-mirror-controller: successfully started 2022-03-07T23:39:34.844 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.829878 I | ceph-client-controller: successfully started 2022-03-07T23:39:34.844 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.829901 I | ceph-filesystem-mirror-controller: successfully started 2022-03-07T23:39:34.844 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.829927 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-07T23:39:34.845 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.829947 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-07T23:39:34.846 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.829965 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-07T23:39:34.847 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.829983 I | ceph-bucket-topic: successfully started 2022-03-07T23:39:34.847 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.829990 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:39:34.847 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.830125 I | ceph-bucket-notification: successfully started 2022-03-07T23:39:34.847 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.830162 I | ceph-bucket-notification: successfully started 2022-03-07T23:39:34.848 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.830186 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-07T23:39:34.848 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.831320 I | operator: starting the controller-runtime manager 2022-03-07T23:39:34.932 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.933286 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-07T23:39:34.933 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.933330 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-07T23:39:34.933 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.933340 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-07T23:39:34.935 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.935766 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-07T23:39:34.949 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.949924 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-07T23:39:34.976 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:34.977143 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-07T23:39:35.025 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:35.026179 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:39:35.026 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:35.026221 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-03-07T23:39:35.026 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:35.026229 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-03-07T23:39:35.029 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:35.027190 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-03-07T23:39:35.029 INFO:tasks.rook.operator.smithi078.stdout:I0307 23:39:35.027240 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-03-07T23:39:35.033 INFO:teuthology.orchestra.run.smithi078.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-07T23:39:35.163 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:35.161945 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:39:35.245 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:35.246627 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-07T23:39:35.436 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:35.437210 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-03-07T23:39:35.601 INFO:teuthology.orchestra.run.smithi078.stderr:+ num_osd=11 2022-03-07T23:39:35.602 INFO:teuthology.orchestra.run.smithi078.stderr:+ sleep 30s 2022-03-07T23:39:35.773 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:35.772900 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one 2022-03-07T23:39:35.971 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:35.971556 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-07T23:39:35.996 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:35.994635 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-03-07T23:39:35.996 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:35.994652 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-03-07T23:39:35.996 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:35.994656 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-03-07T23:39:35.997 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:35.994659 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-03-07T23:39:35.997 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:35.994663 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-03-07T23:39:35.997 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:35.994666 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.5.1" (default) 2022-03-07T23:39:35.997 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:35.994670 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0" (default) 2022-03-07T23:39:35.998 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:35.994674 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-03-07T23:39:35.998 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:35.994677 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-03-07T23:39:35.999 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:35.994680 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1" (default) 2022-03-07T23:39:35.999 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:35.994683 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-03-07T23:39:35.999 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:35.994687 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-03-07T23:39:35.999 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:35.994691 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.2.1" (default) 2022-03-07T23:39:36.000 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:35.994695 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-03-07T23:39:36.000 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:35.994700 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-03-07T23:39:36.002 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:35.994705 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.5.1" 2022-03-07T23:39:36.002 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:35.994762 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-07T23:39:36.002 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:35.994776 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-07T23:39:38.979 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:38.978910 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-07T23:39:41.984 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:41.984613 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-07T23:39:44.895 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.894921 I | ceph-csi: Detected ceph CSI image version: "v3.5.1" 2022-03-07T23:39:44.896 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.894964 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-03-07T23:39:44.896 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.894977 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-07T23:39:44.896 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.894988 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-03-07T23:39:44.896 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.894998 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-07T23:39:44.897 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.895014 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-03-07T23:39:44.897 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.895024 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-07T23:39:44.897 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.895037 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-03-07T23:39:44.897 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.895046 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-07T23:39:44.897 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.895059 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-03-07T23:39:44.900 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.895072 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-07T23:39:44.900 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.895091 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-03-07T23:39:44.901 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.895099 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default) 2022-03-07T23:39:44.901 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.895108 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default) 2022-03-07T23:39:44.901 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.895116 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-03-07T23:39:44.901 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.895124 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-03-07T23:39:44.902 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.895132 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-03-07T23:39:44.902 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.895140 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-03-07T23:39:44.902 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.895149 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-03-07T23:39:44.902 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.895158 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-07T23:39:44.902 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.895167 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-03-07T23:39:44.903 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.895181 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-03-07T23:39:44.903 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.895190 I | ceph-csi: Kubernetes version is 1.23 2022-03-07T23:39:44.903 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.895201 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-03-07T23:39:44.903 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.895209 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-03-07T23:39:44.911 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.911750 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-03-07T23:39:44.915 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.916203 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-03-07T23:39:44.916 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.916222 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-07T23:39:44.916 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.916226 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-03-07T23:39:44.916 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.916229 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-03-07T23:39:44.916 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.916233 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-03-07T23:39:44.917 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.916235 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-03-07T23:39:44.917 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.916239 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-03-07T23:39:44.994 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:44.994055 I | op-k8sutil: batch job rook-ceph-detect-version still exists 2022-03-07T23:39:45.026 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:45.026537 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-03-07T23:39:45.027 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:45.026551 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-07T23:39:45.027 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:45.026555 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-03-07T23:39:45.073 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:45.074047 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-03-07T23:39:45.287 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:45.287250 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-03-07T23:39:45.287 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:45.287281 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-03-07T23:39:45.287 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:45.287295 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-03-07T23:39:45.370 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:45.370854 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-03-07T23:39:45.371 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:45.370881 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-03-07T23:39:45.371 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:45.370888 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-03-07T23:39:45.430 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:45.430879 I | ceph-csi: successfully started CSI CephFS driver 2022-03-07T23:39:45.701 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:45.701591 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-07T23:39:45.705 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:45.706213 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com" 2022-03-07T23:39:45.705 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:45.706233 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-03-07T23:39:45.711 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:45.711876 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com" 2022-03-07T23:39:48.002 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:48.002547 I | op-k8sutil: batch job rook-ceph-detect-version deleted 2022-03-07T23:39:54.030 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:54.027443 I | ceph-spec: detected ceph image version: "17.0.0-11002 quincy" 2022-03-07T23:39:54.030 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:54.027478 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-07T23:39:54.031 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:54.027502 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11002 quincy", pursuing 2022-03-07T23:39:54.037 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:54.037645 I | ceph-spec: detected ceph image version: "17.0.0-11002 quincy" 2022-03-07T23:39:54.038 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:54.037657 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-07T23:39:54.038 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:54.037665 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11002 quincy", pursuing 2022-03-07T23:39:54.042 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:54.038998 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:39:54.060 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:54.060598 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:39:54.218 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:54.216162 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T23:39:54.219 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:54.216445 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T23:39:54.261 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:54.261516 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T23:39:54.264 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:54.262257 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T23:39:54.612 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:54.612560 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11002 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4" 2022-03-07T23:39:54.661 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:54.661093 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11002 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4" 2022-03-07T23:39:54.696 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:54.696298 I | op-mon: start running mons 2022-03-07T23:39:54.721 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:54.722161 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:39:54.725 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:54.722695 E | ceph-spec: failed to update cluster condition to {Type:Progressing Status:True Reason:ClusterProgressing Message:Configuring the Ceph cluster LastHeartbeatTime:2022-03-07 23:39:54.69691876 +0000 UTC m=+433.722542907 LastTransitionTime:2022-03-07 23:39:54.696918527 +0000 UTC m=+433.722542693}. failed to update object "rook-ceph/rook-ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T23:39:55.725 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:55.725070 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.98.222.16:6789","10.96.111.213:6789","10.105.146.196:6789"]}] data:a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 mapping:{"node":{"a":{"Name":"smithi170","Hostname":"smithi170","Address":"172.21.15.170"},"b":{"Name":"smithi166","Hostname":"smithi166","Address":"172.21.15.166"},"c":{"Name":"smithi078","Hostname":"smithi078","Address":"172.21.15.78"}}} maxMonId:2] 2022-03-07T23:39:56.321 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:56.321659 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T23:39:56.325 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:56.321892 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T23:39:57.925 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:57.924916 I | op-mon: targeting the mon count 3 2022-03-07T23:39:57.933 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:57.933202 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T23:39:58.284 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:58.284071 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T23:39:58.284 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:58.284109 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T23:39:58.618 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:58.618193 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T23:39:58.619 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:58.618228 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T23:39:58.952 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:58.952213 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T23:39:58.952 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:58.952250 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-07T23:39:59.286 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:59.284068 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-07T23:39:59.286 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:59.284088 I | op-config: deleting "log file" option from the mon configuration database 2022-03-07T23:39:59.768 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:59.768341 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-07T23:39:59.769 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:59.768373 I | op-mon: checking for basic quorum with existing mons 2022-03-07T23:39:59.913 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:39:59.913237 I | op-mon: mon "a" endpoint is [v2:10.105.146.196:3300,v1:10.105.146.196:6789] 2022-03-07T23:40:00.028 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:00.028729 I | op-mon: mon "b" endpoint is [v2:10.98.222.16:3300,v1:10.98.222.16:6789] 2022-03-07T23:40:00.323 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:00.323634 I | op-mon: mon "c" endpoint is [v2:10.96.111.213:3300,v1:10.96.111.213:6789] 2022-03-07T23:40:00.929 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:00.929618 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.105.146.196:6789","10.98.222.16:6789","10.96.111.213:6789"]}] data:a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 mapping:{"node":{"a":{"Name":"smithi170","Hostname":"smithi170","Address":"172.21.15.170"},"b":{"Name":"smithi166","Hostname":"smithi166","Address":"172.21.15.166"},"c":{"Name":"smithi078","Hostname":"smithi078","Address":"172.21.15.78"}}} maxMonId:2] 2022-03-07T23:40:01.524 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:01.524036 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T23:40:01.529 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:01.524309 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T23:40:01.975 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:01.975446 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-07T23:40:01.992 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:01.992591 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-07T23:40:01.993 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:01.992615 I | op-mon: waiting for mon quorum with [a b c] 2022-03-07T23:40:02.535 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:02.534239 I | op-mon: mons running: [a b c] 2022-03-07T23:40:02.985 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:02.984826 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T23:40:02.993 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:02.993159 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-07T23:40:03.003 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:03.002789 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-07T23:40:03.003 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:03.002814 I | op-mon: waiting for mon quorum with [a b c] 2022-03-07T23:40:03.132 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:03.132280 I | op-mon: mons running: [a b c] 2022-03-07T23:40:03.576 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:03.575759 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T23:40:03.584 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:03.584556 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-07T23:40:03.593 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:03.593719 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-07T23:40:03.594 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:03.593746 I | op-mon: waiting for mon quorum with [a b c] 2022-03-07T23:40:03.734 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:03.733824 I | op-mon: mons running: [a b c] 2022-03-07T23:40:04.138 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:04.137626 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T23:40:04.138 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:04.137650 I | op-mon: mons created: 3 2022-03-07T23:40:04.549 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:04.548385 I | op-mon: waiting for mon quorum with [a b c] 2022-03-07T23:40:04.593 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:04.592872 I | op-mon: mons running: [a b c] 2022-03-07T23:40:05.056 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:05.054227 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T23:40:05.056 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:05.054296 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-07T23:40:05.056 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:05.054393 I | op-mon: start running mons 2022-03-07T23:40:05.063 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:05.063288 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:40:05.500 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:05.499462 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-07T23:40:05.603 INFO:teuthology.orchestra.run.smithi078.stdout:waiting for osd to come back up 2022-03-07T23:40:05.603 INFO:teuthology.orchestra.run.smithi078.stderr:+ '[' 11 '!=' 12 ']' 2022-03-07T23:40:05.604 INFO:teuthology.orchestra.run.smithi078.stderr:+ echo 'waiting for osd to come back up' 2022-03-07T23:40:05.604 INFO:teuthology.orchestra.run.smithi078.stderr:++ toolbox ceph osd stat 2022-03-07T23:40:05.605 INFO:teuthology.orchestra.run.smithi078.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-07T23:40:05.605 INFO:teuthology.orchestra.run.smithi078.stderr:++ cut -f3 -d ' ' 2022-03-07T23:40:05.723 INFO:teuthology.orchestra.run.smithi078.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-07T23:40:05.724 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:05.723490 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.105.146.196:6789","10.98.222.16:6789","10.96.111.213:6789"]}] data:c=10.96.111.213:6789,a=10.105.146.196:6789,b=10.98.222.16:6789 mapping:{"node":{"a":{"Name":"smithi170","Hostname":"smithi170","Address":"172.21.15.170"},"b":{"Name":"smithi166","Hostname":"smithi166","Address":"172.21.15.166"},"c":{"Name":"smithi078","Hostname":"smithi078","Address":"172.21.15.78"}}} maxMonId:2] 2022-03-07T23:40:05.990 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:05.989184 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-07T23:40:06.198 INFO:teuthology.orchestra.run.smithi078.stderr:+ num_osd=11 2022-03-07T23:40:06.198 INFO:teuthology.orchestra.run.smithi078.stderr:+ sleep 30s 2022-03-07T23:40:06.324 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:06.323841 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T23:40:06.328 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:06.324150 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T23:40:06.403 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:06.402895 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-07T23:40:08.925 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:08.924778 I | op-mon: targeting the mon count 3 2022-03-07T23:40:08.933 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:08.933273 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T23:40:09.274 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:09.273688 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T23:40:09.275 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:09.273724 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T23:40:09.524 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:09.523679 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-07T23:40:09.525 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:09.523708 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-07T23:40:09.735 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:09.734497 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T23:40:09.736 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:09.734551 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T23:40:09.971 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:09.970213 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-07T23:40:10.104 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:10.103805 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T23:40:10.105 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:10.103834 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-07T23:40:10.394 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:10.392850 I | cephclient: successfully enabled msgr2 protocol 2022-03-07T23:40:10.394 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:10.392884 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-07T23:40:10.394 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:10.392890 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-07T23:40:10.460 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:10.459389 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-07T23:40:10.460 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:10.459438 I | op-config: deleting "log file" option from the mon configuration database 2022-03-07T23:40:10.838 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:10.837505 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-07T23:40:10.845 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:10.844160 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-07T23:40:10.845 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:10.844184 I | op-mon: checking for basic quorum with existing mons 2022-03-07T23:40:10.894 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:10.893098 I | op-mgr: start running mgr 2022-03-07T23:40:10.894 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:10.893136 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-07T23:40:11.127 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:11.126028 I | op-mon: mon "a" endpoint is [v2:10.105.146.196:3300,v1:10.105.146.196:6789] 2022-03-07T23:40:12.376 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:12.375312 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-07T23:40:12.387 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:12.386505 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-07T23:40:12.391 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:12.389974 I | op-mgr: setting services to point to mgr "a" 2022-03-07T23:40:12.526 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:12.525740 I | op-mon: mon "b" endpoint is [v2:10.98.222.16:3300,v1:10.98.222.16:6789] 2022-03-07T23:40:14.126 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.125772 I | op-mon: mon "c" endpoint is [v2:10.96.111.213:3300,v1:10.96.111.213:6789] 2022-03-07T23:40:14.324 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.323440 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-07T23:40:14.325 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.323575 I | op-mgr: successful modules: balancer 2022-03-07T23:40:14.340 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.339200 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-07T23:40:14.340 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.339271 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-07T23:40:14.420 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.419163 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-07T23:40:14.729 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.725376 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi078-lv-1" 2022-03-07T23:40:14.729 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.725420 I | op-osd: OSD PVC "smithi078-lv-1-data-0mvnbx" already exists 2022-03-07T23:40:14.730 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.725436 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi078-lv-3" 2022-03-07T23:40:14.731 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.725449 I | op-osd: OSD PVC "smithi078-lv-3-data-0wsgp2" already exists 2022-03-07T23:40:14.731 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.725458 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi078-lv-4" 2022-03-07T23:40:14.732 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.725473 I | op-osd: OSD PVC "smithi078-lv-4-data-0mvh7t" already exists 2022-03-07T23:40:14.732 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.725482 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi166-lv-1" 2022-03-07T23:40:14.732 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.725494 I | op-osd: OSD PVC "smithi166-lv-1-data-09wrcv" already exists 2022-03-07T23:40:14.732 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.725504 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi166-lv-2" 2022-03-07T23:40:14.733 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.725515 I | op-osd: OSD PVC "smithi166-lv-2-data-0tz6t9" already exists 2022-03-07T23:40:14.733 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.725528 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi166-lv-3" 2022-03-07T23:40:14.733 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.725539 I | op-osd: OSD PVC "smithi166-lv-3-data-02vsxz" already exists 2022-03-07T23:40:14.733 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.725550 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi166-lv-4" 2022-03-07T23:40:14.734 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.725561 I | op-osd: OSD PVC "smithi166-lv-4-data-0cg6jn" already exists 2022-03-07T23:40:14.734 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.725570 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi170-lv-1" 2022-03-07T23:40:14.734 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.725581 I | op-osd: OSD PVC "smithi170-lv-1-data-076jz6" already exists 2022-03-07T23:40:14.734 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.725591 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi170-lv-2" 2022-03-07T23:40:14.735 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.725602 I | op-osd: OSD PVC "smithi170-lv-2-data-0w6nq6" already exists 2022-03-07T23:40:14.735 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.725619 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi170-lv-3" 2022-03-07T23:40:14.735 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.725640 I | op-osd: OSD PVC "smithi170-lv-3-data-0rc4dr" already exists 2022-03-07T23:40:14.735 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.725655 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi170-lv-4" 2022-03-07T23:40:14.736 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.725674 I | op-osd: OSD PVC "smithi170-lv-4-data-0lsgm7" already exists 2022-03-07T23:40:14.736 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:14.725689 I | op-osd: creating 1 new PVCs for device set "smithi078-lv-2" 2022-03-07T23:40:15.128 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.127389 I | op-osd: successfully provisioned PVC "smithi078-lv-2-data-0mt8gg" 2022-03-07T23:40:15.213 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.209132 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-1-data-0mvnbx" 2022-03-07T23:40:15.213 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.209162 I | op-osd: skipping OSD prepare job creation for PVC "smithi078-lv-1-data-0mvnbx" because OSD daemon using the PVC already exists 2022-03-07T23:40:15.214 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.209169 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-3-data-0wsgp2" 2022-03-07T23:40:15.214 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.209174 I | op-osd: skipping OSD prepare job creation for PVC "smithi078-lv-3-data-0wsgp2" because OSD daemon using the PVC already exists 2022-03-07T23:40:15.214 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.209179 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-4-data-0mvh7t" 2022-03-07T23:40:15.214 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.209184 I | op-osd: skipping OSD prepare job creation for PVC "smithi078-lv-4-data-0mvh7t" because OSD daemon using the PVC already exists 2022-03-07T23:40:15.214 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.209190 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-1-data-09wrcv" 2022-03-07T23:40:15.215 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.209194 I | op-osd: skipping OSD prepare job creation for PVC "smithi166-lv-1-data-09wrcv" because OSD daemon using the PVC already exists 2022-03-07T23:40:15.215 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.209199 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-2-data-0tz6t9" 2022-03-07T23:40:15.215 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.209204 I | op-osd: skipping OSD prepare job creation for PVC "smithi166-lv-2-data-0tz6t9" because OSD daemon using the PVC already exists 2022-03-07T23:40:15.215 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.209210 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-3-data-02vsxz" 2022-03-07T23:40:15.216 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.209215 I | op-osd: skipping OSD prepare job creation for PVC "smithi166-lv-3-data-02vsxz" because OSD daemon using the PVC already exists 2022-03-07T23:40:15.216 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.209219 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-4-data-0cg6jn" 2022-03-07T23:40:15.216 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.209224 I | op-osd: skipping OSD prepare job creation for PVC "smithi166-lv-4-data-0cg6jn" because OSD daemon using the PVC already exists 2022-03-07T23:40:15.216 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.209229 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-1-data-076jz6" 2022-03-07T23:40:15.216 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.209235 I | op-osd: skipping OSD prepare job creation for PVC "smithi170-lv-1-data-076jz6" because OSD daemon using the PVC already exists 2022-03-07T23:40:15.217 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.209240 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-2-data-0w6nq6" 2022-03-07T23:40:15.217 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.209245 I | op-osd: skipping OSD prepare job creation for PVC "smithi170-lv-2-data-0w6nq6" because OSD daemon using the PVC already exists 2022-03-07T23:40:15.217 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.209250 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-3-data-0rc4dr" 2022-03-07T23:40:15.219 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.209255 I | op-osd: skipping OSD prepare job creation for PVC "smithi170-lv-3-data-0rc4dr" because OSD daemon using the PVC already exists 2022-03-07T23:40:15.220 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.209260 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-4-data-0lsgm7" 2022-03-07T23:40:15.220 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.209265 I | op-osd: skipping OSD prepare job creation for PVC "smithi170-lv-4-data-0lsgm7" because OSD daemon using the PVC already exists 2022-03-07T23:40:15.220 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.209270 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-2-data-0mt8gg" 2022-03-07T23:40:15.324 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.323492 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.105.146.196:6789","10.98.222.16:6789","10.96.111.213:6789"]}] data:a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 mapping:{"node":{"a":{"Name":"smithi170","Hostname":"smithi170","Address":"172.21.15.170"},"b":{"Name":"smithi166","Hostname":"smithi166","Address":"172.21.15.166"},"c":{"Name":"smithi078","Hostname":"smithi078","Address":"172.21.15.78"}}} maxMonId:2] 2022-03-07T23:40:15.940 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.939686 I | op-osd: started OSD provisioning job for PVC "smithi078-lv-2-data-0mt8gg" 2022-03-07T23:40:15.941 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.939719 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-07T23:40:15.941 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:15.939727 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-07T23:40:16.046 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:16.045026 I | op-mgr: successful modules: dashboard 2022-03-07T23:40:16.046 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:16.045060 I | op-mgr: successful modules: prometheus 2022-03-07T23:40:16.048 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:16.045594 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-07T23:40:16.325 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:16.324123 I | op-osd: OSD orchestration status for PVC smithi078-lv-2-data-0mt8gg is "starting" 2022-03-07T23:40:16.427 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:16.426136 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:16.433 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:16.432785 I | op-osd: updating OSD 1 on PVC "smithi166-lv-1-data-09wrcv" 2022-03-07T23:40:16.434 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:16.432818 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-1-data-09wrcv" 2022-03-07T23:40:16.522 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:16.520798 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T23:40:16.531 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:16.521101 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T23:40:16.623 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:16.622440 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:16.629 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:16.628163 I | op-osd: updating OSD 10 on PVC "smithi078-lv-4-data-0mvh7t" 2022-03-07T23:40:16.629 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:16.628197 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-4-data-0mvh7t" 2022-03-07T23:40:16.794 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:16.793177 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:16.800 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:16.799037 I | op-osd: updating OSD 11 on PVC "smithi078-lv-3-data-0wsgp2" 2022-03-07T23:40:16.800 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:16.799067 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-3-data-0wsgp2" 2022-03-07T23:40:16.979 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:16.977804 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:17.234 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:17.233453 I | op-osd: updating OSD 2 on PVC "smithi170-lv-1-data-076jz6" 2022-03-07T23:40:17.235 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:17.233501 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-1-data-076jz6" 2022-03-07T23:40:17.433 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:17.432424 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-07T23:40:17.666 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:17.665082 I | op-mgr: successful modules: orchestrator modules 2022-03-07T23:40:17.840 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:17.838890 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-07T23:40:17.840 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:17.838920 I | op-mon: waiting for mon quorum with [a b c] 2022-03-07T23:40:17.874 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:17.873779 I | op-mon: mons running: [a b c] 2022-03-07T23:40:18.300 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:18.298059 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T23:40:18.338 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:18.337089 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:18.449 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:18.448413 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-07T23:40:18.640 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:18.638730 I | op-osd: updating OSD 3 on PVC "smithi166-lv-2-data-0tz6t9" 2022-03-07T23:40:18.640 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:18.638770 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-2-data-0tz6t9" 2022-03-07T23:40:18.900 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:18.899326 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-07T23:40:18.901 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:18.899365 I | op-mon: waiting for mon quorum with [a b c] 2022-03-07T23:40:18.935 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:18.932565 I | op-mon: mons running: [a b c] 2022-03-07T23:40:19.355 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:19.353936 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T23:40:19.433 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:19.432109 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-07T23:40:19.775 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:19.774444 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-07T23:40:19.776 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:19.774468 I | op-mon: waiting for mon quorum with [a b c] 2022-03-07T23:40:19.983 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:19.981849 I | op-mon: mons running: [a b c] 2022-03-07T23:40:20.034 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:20.033176 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:20.090 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:20.088993 I | op-osd: updating OSD 4 on PVC "smithi170-lv-2-data-0w6nq6" 2022-03-07T23:40:20.090 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:20.089026 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-2-data-0w6nq6" 2022-03-07T23:40:20.425 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:20.423654 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T23:40:20.425 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:20.423682 I | op-mon: mons created: 3 2022-03-07T23:40:20.740 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:20.739000 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:20.833 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:20.832034 I | op-mon: waiting for mon quorum with [a b c] 2022-03-07T23:40:20.936 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:20.932355 I | op-osd: updating OSD 5 on PVC "smithi166-lv-3-data-02vsxz" 2022-03-07T23:40:20.936 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:20.932397 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-3-data-02vsxz" 2022-03-07T23:40:20.955 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:20.953968 I | op-mon: mons running: [a b c] 2022-03-07T23:40:21.385 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:21.384053 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T23:40:21.386 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:21.384123 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-07T23:40:21.537 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:21.535458 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:21.774 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:21.772880 I | op-osd: updating OSD 6 on PVC "smithi170-lv-4-data-0lsgm7" 2022-03-07T23:40:21.774 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:21.772920 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-4-data-0lsgm7" 2022-03-07T23:40:21.790 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:21.788794 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-07T23:40:22.242 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:22.240855 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-07T23:40:22.533 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:22.531728 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:22.539 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:22.537856 I | op-osd: updating OSD 7 on PVC "smithi166-lv-4-data-0cg6jn" 2022-03-07T23:40:22.539 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:22.537884 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-4-data-0cg6jn" 2022-03-07T23:40:22.900 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:22.898149 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-07T23:40:23.147 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:23.146002 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:23.237 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:23.235583 I | op-osd: updating OSD 8 on PVC "smithi170-lv-3-data-0rc4dr" 2022-03-07T23:40:23.237 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:23.235617 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-3-data-0rc4dr" 2022-03-07T23:40:24.104 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:24.102723 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-07T23:40:24.104 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:24.102758 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-07T23:40:24.203 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:24.201466 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:24.211 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:24.207347 I | op-osd: updating OSD 9 on PVC "smithi078-lv-1-data-0mvnbx" 2022-03-07T23:40:24.212 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:24.207386 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-1-data-0mvnbx" 2022-03-07T23:40:24.557 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:24.555678 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-07T23:40:24.971 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:24.968973 I | cephclient: successfully enabled msgr2 protocol 2022-03-07T23:40:24.971 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:24.969012 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-07T23:40:24.972 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:24.969019 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-07T23:40:25.398 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:25.396935 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-07T23:40:25.533 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:25.531750 I | op-mgr: start running mgr 2022-03-07T23:40:25.533 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:25.531781 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-07T23:40:26.055 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.054063 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-07T23:40:26.066 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.065268 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-07T23:40:26.073 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.068575 I | op-mgr: setting services to point to mgr "a" 2022-03-07T23:40:26.453 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.451687 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-07T23:40:26.453 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.451870 I | op-mgr: successful modules: balancer 2022-03-07T23:40:26.516 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.514243 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-07T23:40:26.516 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.514279 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-07T23:40:26.544 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.543027 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-07T23:40:26.550 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.547220 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi078-lv-1" 2022-03-07T23:40:26.551 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.547246 I | op-osd: OSD PVC "smithi078-lv-1-data-0mvnbx" already exists 2022-03-07T23:40:26.551 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.547253 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi078-lv-3" 2022-03-07T23:40:26.551 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.547259 I | op-osd: OSD PVC "smithi078-lv-3-data-0wsgp2" already exists 2022-03-07T23:40:26.551 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.547265 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi078-lv-4" 2022-03-07T23:40:26.551 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.547273 I | op-osd: OSD PVC "smithi078-lv-4-data-0mvh7t" already exists 2022-03-07T23:40:26.552 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.547280 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi166-lv-1" 2022-03-07T23:40:26.552 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.547290 I | op-osd: OSD PVC "smithi166-lv-1-data-09wrcv" already exists 2022-03-07T23:40:26.552 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.547297 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi166-lv-2" 2022-03-07T23:40:26.552 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.547305 I | op-osd: OSD PVC "smithi166-lv-2-data-0tz6t9" already exists 2022-03-07T23:40:26.553 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.547311 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi166-lv-3" 2022-03-07T23:40:26.553 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.547319 I | op-osd: OSD PVC "smithi166-lv-3-data-02vsxz" already exists 2022-03-07T23:40:26.553 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.547329 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi166-lv-4" 2022-03-07T23:40:26.553 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.547338 I | op-osd: OSD PVC "smithi166-lv-4-data-0cg6jn" already exists 2022-03-07T23:40:26.553 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.547348 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi170-lv-1" 2022-03-07T23:40:26.554 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.547358 I | op-osd: OSD PVC "smithi170-lv-1-data-076jz6" already exists 2022-03-07T23:40:26.554 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.547366 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi170-lv-2" 2022-03-07T23:40:26.554 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.547375 I | op-osd: OSD PVC "smithi170-lv-2-data-0w6nq6" already exists 2022-03-07T23:40:26.554 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.547385 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi170-lv-3" 2022-03-07T23:40:26.555 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.547396 I | op-osd: OSD PVC "smithi170-lv-3-data-0rc4dr" already exists 2022-03-07T23:40:26.555 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.547430 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi170-lv-4" 2022-03-07T23:40:26.555 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.547445 I | op-osd: OSD PVC "smithi170-lv-4-data-0lsgm7" already exists 2022-03-07T23:40:26.555 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.547453 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi078-lv-2" 2022-03-07T23:40:26.556 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.547462 I | op-osd: OSD PVC "smithi078-lv-2-data-0mt8gg" already exists 2022-03-07T23:40:26.580 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.574710 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-1-data-0mvnbx" 2022-03-07T23:40:26.581 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.574736 I | op-osd: skipping OSD prepare job creation for PVC "smithi078-lv-1-data-0mvnbx" because OSD daemon using the PVC already exists 2022-03-07T23:40:26.581 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.574741 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-3-data-0wsgp2" 2022-03-07T23:40:26.581 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.574745 I | op-osd: skipping OSD prepare job creation for PVC "smithi078-lv-3-data-0wsgp2" because OSD daemon using the PVC already exists 2022-03-07T23:40:26.582 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.574748 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-4-data-0mvh7t" 2022-03-07T23:40:26.582 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.574752 I | op-osd: skipping OSD prepare job creation for PVC "smithi078-lv-4-data-0mvh7t" because OSD daemon using the PVC already exists 2022-03-07T23:40:26.582 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.574756 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-1-data-09wrcv" 2022-03-07T23:40:26.582 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.574759 I | op-osd: skipping OSD prepare job creation for PVC "smithi166-lv-1-data-09wrcv" because OSD daemon using the PVC already exists 2022-03-07T23:40:26.583 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.574762 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-2-data-0tz6t9" 2022-03-07T23:40:26.583 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.574766 I | op-osd: skipping OSD prepare job creation for PVC "smithi166-lv-2-data-0tz6t9" because OSD daemon using the PVC already exists 2022-03-07T23:40:26.583 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.574771 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-3-data-02vsxz" 2022-03-07T23:40:26.583 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.574777 I | op-osd: skipping OSD prepare job creation for PVC "smithi166-lv-3-data-02vsxz" because OSD daemon using the PVC already exists 2022-03-07T23:40:26.584 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.574782 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-4-data-0cg6jn" 2022-03-07T23:40:26.584 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.574788 I | op-osd: skipping OSD prepare job creation for PVC "smithi166-lv-4-data-0cg6jn" because OSD daemon using the PVC already exists 2022-03-07T23:40:26.584 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.574794 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-1-data-076jz6" 2022-03-07T23:40:26.584 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.574801 I | op-osd: skipping OSD prepare job creation for PVC "smithi170-lv-1-data-076jz6" because OSD daemon using the PVC already exists 2022-03-07T23:40:26.585 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.574806 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-2-data-0w6nq6" 2022-03-07T23:40:26.585 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.574812 I | op-osd: skipping OSD prepare job creation for PVC "smithi170-lv-2-data-0w6nq6" because OSD daemon using the PVC already exists 2022-03-07T23:40:26.585 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.574818 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-3-data-0rc4dr" 2022-03-07T23:40:26.585 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.574824 I | op-osd: skipping OSD prepare job creation for PVC "smithi170-lv-3-data-0rc4dr" because OSD daemon using the PVC already exists 2022-03-07T23:40:26.585 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.574833 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-4-data-0lsgm7" 2022-03-07T23:40:26.586 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.574838 I | op-osd: skipping OSD prepare job creation for PVC "smithi170-lv-4-data-0lsgm7" because OSD daemon using the PVC already exists 2022-03-07T23:40:26.586 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.574845 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-2-data-0mt8gg" 2022-03-07T23:40:26.587 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.583263 I | op-k8sutil: Found previous job rook-ceph-osd-prepare-smithi078-lv-2-data-0mt8gg. Status={Conditions:[] StartTime:2022-03-07 23:40:15 +0000 UTC CompletionTime: Active:1 Succeeded:0 Failed:0 CompletedIndexes: UncountedTerminatedPods:&UncountedTerminatedPods{Succeeded:[],Failed:[],}} 2022-03-07T23:40:26.587 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.583289 I | op-osd: started OSD provisioning job for PVC "smithi078-lv-2-data-0mt8gg" 2022-03-07T23:40:26.587 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.583297 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-07T23:40:26.588 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.583301 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-07T23:40:26.588 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.585618 I | op-osd: OSD orchestration status for PVC smithi078-lv-2-data-0mt8gg is "starting" 2022-03-07T23:40:26.688 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.686804 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:26.694 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.692973 I | op-osd: updating OSD 1 on PVC "smithi166-lv-1-data-09wrcv" 2022-03-07T23:40:26.694 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.693001 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-1-data-09wrcv" 2022-03-07T23:40:26.701 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.700764 I | op-osd: OSD orchestration status for node smithi078-lv-2-data-0mt8gg is "orchestrating" 2022-03-07T23:40:26.840 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.839000 I | op-osd: OSD orchestration status for node smithi078-lv-2-data-0mt8gg is "orchestrating" 2022-03-07T23:40:26.942 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.939657 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:26.946 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.945450 I | op-osd: updating OSD 10 on PVC "smithi078-lv-4-data-0mvh7t" 2022-03-07T23:40:26.947 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:26.945485 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-4-data-0mvh7t" 2022-03-07T23:40:27.098 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:27.095742 I | op-osd: OSD orchestration status for PVC smithi078-lv-2-data-0mt8gg is "orchestrating" 2022-03-07T23:40:27.238 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:27.236615 I | op-osd: OSD orchestration status for PVC smithi078-lv-2-data-0mt8gg is "orchestrating" 2022-03-07T23:40:27.338 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:27.337002 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:27.434 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:27.432552 I | op-osd: updating OSD 11 on PVC "smithi078-lv-3-data-0wsgp2" 2022-03-07T23:40:27.435 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:27.432588 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-3-data-0wsgp2" 2022-03-07T23:40:28.104 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:28.102731 I | op-mgr: successful modules: prometheus 2022-03-07T23:40:28.107 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:28.106203 I | op-mgr: successful modules: dashboard 2022-03-07T23:40:28.111 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:28.107645 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-07T23:40:28.141 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:28.140204 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:28.232 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:28.231346 I | op-osd: updating OSD 2 on PVC "smithi170-lv-1-data-076jz6" 2022-03-07T23:40:28.233 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:28.231366 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-1-data-076jz6" 2022-03-07T23:40:28.939 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:28.937451 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:29.035 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:29.033618 I | op-osd: updating OSD 3 on PVC "smithi166-lv-2-data-0tz6t9" 2022-03-07T23:40:29.035 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:29.033653 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-2-data-0tz6t9" 2022-03-07T23:40:29.590 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:29.588232 I | op-mgr: successful modules: orchestrator modules 2022-03-07T23:40:29.740 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:29.738938 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:29.834 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:29.832255 I | op-osd: updating OSD 4 on PVC "smithi170-lv-2-data-0w6nq6" 2022-03-07T23:40:29.834 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:29.832302 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-2-data-0w6nq6" 2022-03-07T23:40:30.553 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:30.551378 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:30.634 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:30.632529 I | op-osd: updating OSD 5 on PVC "smithi166-lv-3-data-02vsxz" 2022-03-07T23:40:30.634 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:30.632565 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-3-data-02vsxz" 2022-03-07T23:40:31.340 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:31.338865 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:31.434 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:31.432518 I | op-osd: updating OSD 6 on PVC "smithi170-lv-4-data-0lsgm7" 2022-03-07T23:40:31.434 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:31.432563 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-4-data-0lsgm7" 2022-03-07T23:40:32.171 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:32.169189 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:32.234 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:32.232729 I | op-osd: updating OSD 7 on PVC "smithi166-lv-4-data-0cg6jn" 2022-03-07T23:40:32.234 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:32.232763 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-4-data-0cg6jn" 2022-03-07T23:40:32.940 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:32.938603 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:33.035 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:33.033845 I | op-osd: updating OSD 8 on PVC "smithi170-lv-3-data-0rc4dr" 2022-03-07T23:40:33.036 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:33.033883 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-3-data-0rc4dr" 2022-03-07T23:40:33.275 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:33.274032 I | op-osd: OSD orchestration status for PVC smithi078-lv-2-data-0mt8gg is "completed" 2022-03-07T23:40:33.276 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:33.274066 I | op-osd: creating OSD 0 on PVC "smithi078-lv-2-data-0mt8gg" 2022-03-07T23:40:33.276 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:33.274083 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-2-data-0mt8gg" 2022-03-07T23:40:33.854 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:33.851926 I | op-osd: OSD orchestration status for PVC smithi078-lv-2-data-0mt8gg is "completed" 2022-03-07T23:40:33.858 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:33.852803 I | op-osd: creating OSD 0 on PVC "smithi078-lv-2-data-0mt8gg" 2022-03-07T23:40:33.858 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:33.852977 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-2-data-0mt8gg" 2022-03-07T23:40:34.494 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:34.492575 E | op-osd: failed to create OSD 0 on PVC "smithi078-lv-2-data-0mt8gg": failed to create deployment for OSD 0 on PVC "smithi078-lv-2-data-0mt8gg": deployments.apps "rook-ceph-osd-0" already exists 2022-03-07T23:40:34.591 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:34.589598 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-07T23:40:34.592 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:34.589630 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-07T23:40:34.602 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:34.599963 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:34.671 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:34.669658 I | op-osd: updating OSD 9 on PVC "smithi078-lv-1-data-0mvnbx" 2022-03-07T23:40:34.671 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:34.669690 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-1-data-0mvnbx" 2022-03-07T23:40:35.661 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:35.659350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph osds: 1 failures encountered while running osds on nodes in namespace "rook-ceph". 2022-03-07T23:40:35.662 INFO:tasks.rook.operator.smithi078.stdout:failed to create OSD 0 on PVC "smithi078-lv-2-data-0mt8gg": failed to create deployment for OSD 0 on PVC "smithi078-lv-2-data-0mt8gg": deployments.apps "rook-ceph-osd-0" already exists 2022-03-07T23:40:35.666 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:35.665554 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-03-07T23:40:35.853 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:35.851143 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:40:35.869 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:35.867987 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-07T23:40:36.201 INFO:teuthology.orchestra.run.smithi078.stdout:waiting for osd to come back up 2022-03-07T23:40:36.202 INFO:teuthology.orchestra.run.smithi078.stderr:+ '[' 11 '!=' 12 ']' 2022-03-07T23:40:36.202 INFO:teuthology.orchestra.run.smithi078.stderr:+ echo 'waiting for osd to come back up' 2022-03-07T23:40:36.202 INFO:teuthology.orchestra.run.smithi078.stderr:++ toolbox ceph osd stat 2022-03-07T23:40:36.202 INFO:teuthology.orchestra.run.smithi078.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-07T23:40:36.203 INFO:teuthology.orchestra.run.smithi078.stderr:++ cut -f3 -d ' ' 2022-03-07T23:40:36.876 INFO:teuthology.orchestra.run.smithi078.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-07T23:40:37.483 INFO:teuthology.orchestra.run.smithi078.stderr:+ num_osd=11 2022-03-07T23:40:37.483 INFO:teuthology.orchestra.run.smithi078.stderr:+ sleep 30s 2022-03-07T23:40:43.142 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:43.140640 I | ceph-spec: detected ceph image version: "17.0.0-11002 quincy" 2022-03-07T23:40:43.143 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:43.140673 I | ceph-cluster-controller: validating ceph version from provided image 2022-03-07T23:40:43.143 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:43.140688 W | ceph-cluster-controller: unsupported ceph version detected: "17.0.0-11002 quincy", pursuing 2022-03-07T23:40:43.161 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:43.160010 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:40:43.276 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:43.274978 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T23:40:43.277 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:43.275100 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T23:40:43.696 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:43.694577 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-11002 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4" 2022-03-07T23:40:43.831 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:43.827779 I | op-mon: start running mons 2022-03-07T23:40:43.930 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:43.928184 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:40:43.987 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:43.985896 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.105.146.196:6789","10.98.222.16:6789","10.96.111.213:6789"]}] data:a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 mapping:{"node":{"a":{"Name":"smithi170","Hostname":"smithi170","Address":"172.21.15.170"},"b":{"Name":"smithi166","Hostname":"smithi166","Address":"172.21.15.166"},"c":{"Name":"smithi078","Hostname":"smithi078","Address":"172.21.15.78"}}} maxMonId:2] 2022-03-07T23:40:44.208 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:44.205655 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T23:40:44.316 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:44.314182 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T23:40:45.777 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:45.775533 I | op-mon: targeting the mon count 3 2022-03-07T23:40:45.785 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:45.783788 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T23:40:46.248 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:46.246162 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-03-07T23:40:46.248 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:46.246198 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T23:40:46.608 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:46.605998 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-03-07T23:40:46.608 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:46.606034 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T23:40:46.951 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:46.948780 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-03-07T23:40:46.951 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:46.948814 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-03-07T23:40:47.285 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:47.283098 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-03-07T23:40:47.285 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:47.283131 I | op-config: deleting "log file" option from the mon configuration database 2022-03-07T23:40:47.793 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:47.790733 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-03-07T23:40:47.793 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:47.790766 I | op-mon: checking for basic quorum with existing mons 2022-03-07T23:40:47.913 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:47.911137 I | op-mon: mon "b" endpoint is [v2:10.98.222.16:3300,v1:10.98.222.16:6789] 2022-03-07T23:40:48.068 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:48.066899 I | op-mon: mon "c" endpoint is [v2:10.96.111.213:3300,v1:10.96.111.213:6789] 2022-03-07T23:40:48.679 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:48.677386 I | op-mon: mon "a" endpoint is [v2:10.105.146.196:3300,v1:10.105.146.196:6789] 2022-03-07T23:40:48.819 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:48.817448 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.105.146.196:6789","10.98.222.16:6789","10.96.111.213:6789"]}] data:a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 mapping:{"node":{"a":{"Name":"smithi170","Hostname":"smithi170","Address":"172.21.15.170"},"b":{"Name":"smithi166","Hostname":"smithi166","Address":"172.21.15.166"},"c":{"Name":"smithi078","Hostname":"smithi078","Address":"172.21.15.78"}}} maxMonId:2] 2022-03-07T23:40:49.486 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:49.484525 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-03-07T23:40:49.487 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:49.484804 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-03-07T23:40:50.178 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:50.176199 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-03-07T23:40:50.189 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:50.187712 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-03-07T23:40:50.190 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:50.187738 I | op-mon: waiting for mon quorum with [b c a] 2022-03-07T23:40:50.407 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:50.405881 I | op-mon: mons running: [b c a] 2022-03-07T23:40:50.821 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:50.819117 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T23:40:50.827 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:50.825297 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed 2022-03-07T23:40:50.860 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:50.857313 I | op-k8sutil: deployment "rook-ceph-mon-c" did not change, nothing to update 2022-03-07T23:40:50.861 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:50.857338 I | op-mon: waiting for mon quorum with [b c a] 2022-03-07T23:40:51.006 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:51.004856 I | op-mon: mons running: [b c a] 2022-03-07T23:40:51.451 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:51.449587 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T23:40:51.460 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:51.459007 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-03-07T23:40:51.471 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:51.469393 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-03-07T23:40:51.472 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:51.469424 I | op-mon: waiting for mon quorum with [b c a] 2022-03-07T23:40:51.606 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:51.604095 I | op-mon: mons running: [b c a] 2022-03-07T23:40:52.019 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:52.017458 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T23:40:52.020 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:52.017489 I | op-mon: mons created: 3 2022-03-07T23:40:52.434 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:52.432240 I | op-mon: waiting for mon quorum with [b c a] 2022-03-07T23:40:52.467 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:52.465580 I | op-mon: mons running: [b c a] 2022-03-07T23:40:52.870 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:52.868650 I | op-mon: Monitors in quorum: [a b c] 2022-03-07T23:40:52.871 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:52.868692 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-03-07T23:40:53.290 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:53.288434 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-03-07T23:40:53.763 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:53.760839 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-03-07T23:40:54.180 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:54.178478 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-03-07T23:40:54.679 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:54.676961 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-03-07T23:40:54.679 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:54.676988 I | cephclient: getting or creating ceph auth key "client.crash" 2022-03-07T23:40:55.125 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:55.123499 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-03-07T23:40:55.550 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:55.548069 I | cephclient: successfully enabled msgr2 protocol 2022-03-07T23:40:55.551 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:55.548117 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-03-07T23:40:55.551 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:55.548126 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-03-07T23:40:56.047 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.045374 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-03-07T23:40:56.084 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.082798 I | op-mgr: start running mgr 2022-03-07T23:40:56.085 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.082828 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-03-07T23:40:56.557 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.555074 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed 2022-03-07T23:40:56.566 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.564808 I | op-k8sutil: deployment "rook-ceph-mgr-a" did not change, nothing to update 2022-03-07T23:40:56.570 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.568411 I | op-mgr: setting services to point to mgr "a" 2022-03-07T23:40:56.675 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.672914 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-03-07T23:40:56.675 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.672997 I | op-mgr: successful modules: balancer 2022-03-07T23:40:56.689 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.687035 I | op-osd: start running osds in namespace "rook-ceph" 2022-03-07T23:40:56.689 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.687065 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-03-07T23:40:56.748 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.746740 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-03-07T23:40:56.863 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.859683 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi078-lv-1" 2022-03-07T23:40:56.864 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.859719 I | op-osd: OSD PVC "smithi078-lv-1-data-0mvnbx" already exists 2022-03-07T23:40:56.864 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.859731 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi078-lv-3" 2022-03-07T23:40:56.864 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.859742 I | op-osd: OSD PVC "smithi078-lv-3-data-0wsgp2" already exists 2022-03-07T23:40:56.865 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.859749 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi078-lv-4" 2022-03-07T23:40:56.865 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.859757 I | op-osd: OSD PVC "smithi078-lv-4-data-0mvh7t" already exists 2022-03-07T23:40:56.865 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.859764 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi166-lv-1" 2022-03-07T23:40:56.865 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.859772 I | op-osd: OSD PVC "smithi166-lv-1-data-09wrcv" already exists 2022-03-07T23:40:56.866 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.859778 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi166-lv-2" 2022-03-07T23:40:56.866 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.859786 I | op-osd: OSD PVC "smithi166-lv-2-data-0tz6t9" already exists 2022-03-07T23:40:56.866 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.859793 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi166-lv-3" 2022-03-07T23:40:56.866 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.859801 I | op-osd: OSD PVC "smithi166-lv-3-data-02vsxz" already exists 2022-03-07T23:40:56.866 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.859807 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi166-lv-4" 2022-03-07T23:40:56.867 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.859815 I | op-osd: OSD PVC "smithi166-lv-4-data-0cg6jn" already exists 2022-03-07T23:40:56.867 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.859821 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi170-lv-1" 2022-03-07T23:40:56.867 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.859829 I | op-osd: OSD PVC "smithi170-lv-1-data-076jz6" already exists 2022-03-07T23:40:56.867 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.859837 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi170-lv-2" 2022-03-07T23:40:56.868 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.859845 I | op-osd: OSD PVC "smithi170-lv-2-data-0w6nq6" already exists 2022-03-07T23:40:56.868 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.859853 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi170-lv-3" 2022-03-07T23:40:56.868 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.859862 I | op-osd: OSD PVC "smithi170-lv-3-data-0rc4dr" already exists 2022-03-07T23:40:56.868 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.859869 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi170-lv-4" 2022-03-07T23:40:56.868 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.859877 I | op-osd: OSD PVC "smithi170-lv-4-data-0lsgm7" already exists 2022-03-07T23:40:56.869 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.859884 I | op-osd: verifying PVCs exist for 1 OSDs in device set "smithi078-lv-2" 2022-03-07T23:40:56.869 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.859892 I | op-osd: OSD PVC "smithi078-lv-2-data-0mt8gg" already exists 2022-03-07T23:40:56.894 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.889806 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-1-data-0mvnbx" 2022-03-07T23:40:56.895 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.889835 I | op-osd: skipping OSD prepare job creation for PVC "smithi078-lv-1-data-0mvnbx" because OSD daemon using the PVC already exists 2022-03-07T23:40:56.895 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.889841 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-3-data-0wsgp2" 2022-03-07T23:40:56.895 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.889846 I | op-osd: skipping OSD prepare job creation for PVC "smithi078-lv-3-data-0wsgp2" because OSD daemon using the PVC already exists 2022-03-07T23:40:56.896 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.889851 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-4-data-0mvh7t" 2022-03-07T23:40:56.896 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.889855 I | op-osd: skipping OSD prepare job creation for PVC "smithi078-lv-4-data-0mvh7t" because OSD daemon using the PVC already exists 2022-03-07T23:40:56.896 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.889861 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-1-data-09wrcv" 2022-03-07T23:40:56.896 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.889865 I | op-osd: skipping OSD prepare job creation for PVC "smithi166-lv-1-data-09wrcv" because OSD daemon using the PVC already exists 2022-03-07T23:40:56.897 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.889870 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-2-data-0tz6t9" 2022-03-07T23:40:56.897 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.889874 I | op-osd: skipping OSD prepare job creation for PVC "smithi166-lv-2-data-0tz6t9" because OSD daemon using the PVC already exists 2022-03-07T23:40:56.897 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.889879 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-3-data-02vsxz" 2022-03-07T23:40:56.898 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.889884 I | op-osd: skipping OSD prepare job creation for PVC "smithi166-lv-3-data-02vsxz" because OSD daemon using the PVC already exists 2022-03-07T23:40:56.898 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.889889 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-4-data-0cg6jn" 2022-03-07T23:40:56.898 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.889893 I | op-osd: skipping OSD prepare job creation for PVC "smithi166-lv-4-data-0cg6jn" because OSD daemon using the PVC already exists 2022-03-07T23:40:56.898 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.889898 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-1-data-076jz6" 2022-03-07T23:40:56.899 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.889902 I | op-osd: skipping OSD prepare job creation for PVC "smithi170-lv-1-data-076jz6" because OSD daemon using the PVC already exists 2022-03-07T23:40:56.899 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.889907 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-2-data-0w6nq6" 2022-03-07T23:40:56.899 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.889912 I | op-osd: skipping OSD prepare job creation for PVC "smithi170-lv-2-data-0w6nq6" because OSD daemon using the PVC already exists 2022-03-07T23:40:56.899 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.889917 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-3-data-0rc4dr" 2022-03-07T23:40:56.899 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.889921 I | op-osd: skipping OSD prepare job creation for PVC "smithi170-lv-3-data-0rc4dr" because OSD daemon using the PVC already exists 2022-03-07T23:40:56.900 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.889926 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-4-data-0lsgm7" 2022-03-07T23:40:56.900 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.889931 I | op-osd: skipping OSD prepare job creation for PVC "smithi170-lv-4-data-0lsgm7" because OSD daemon using the PVC already exists 2022-03-07T23:40:56.900 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.889935 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-2-data-0mt8gg" 2022-03-07T23:40:56.900 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.889940 I | op-osd: skipping OSD prepare job creation for PVC "smithi078-lv-2-data-0mt8gg" because OSD daemon using the PVC already exists 2022-03-07T23:40:56.901 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.889944 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-03-07T23:40:56.901 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:56.889948 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-03-07T23:40:57.160 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:57.157951 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:57.165 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:57.163088 I | op-osd: updating OSD 0 on PVC "smithi078-lv-2-data-0mt8gg" 2022-03-07T23:40:57.165 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:57.163123 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-2-data-0mt8gg" 2022-03-07T23:40:57.415 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:57.413209 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:57.428 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:57.421729 I | op-osd: updating OSD 1 on PVC "smithi166-lv-1-data-09wrcv" 2022-03-07T23:40:57.428 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:57.421786 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-1-data-09wrcv" 2022-03-07T23:40:57.582 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:57.579890 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:57.589 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:57.586735 I | op-osd: updating OSD 10 on PVC "smithi078-lv-4-data-0mvh7t" 2022-03-07T23:40:57.589 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:57.586792 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-4-data-0mvh7t" 2022-03-07T23:40:58.037 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:58.035235 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:58.130 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:58.128150 I | op-osd: updating OSD 11 on PVC "smithi078-lv-3-data-0wsgp2" 2022-03-07T23:40:58.130 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:58.128175 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-3-data-0wsgp2" 2022-03-07T23:40:58.392 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:58.390073 I | op-mgr: successful modules: dashboard 2022-03-07T23:40:58.395 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:58.391232 I | op-mgr: successful modules: prometheus 2022-03-07T23:40:58.395 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:58.392047 I | op-mgr: successful modules: mgr module(s) from the spec 2022-03-07T23:40:58.841 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:58.839445 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:58.931 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:58.929289 I | op-osd: updating OSD 2 on PVC "smithi170-lv-1-data-076jz6" 2022-03-07T23:40:58.932 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:58.929325 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-1-data-076jz6" 2022-03-07T23:40:59.638 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:59.636181 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:40:59.733 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:59.730041 I | op-osd: updating OSD 3 on PVC "smithi166-lv-2-data-0tz6t9" 2022-03-07T23:40:59.733 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:40:59.730073 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-2-data-0tz6t9" 2022-03-07T23:41:00.466 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:00.462661 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:41:00.487 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:00.484784 I | op-mgr: successful modules: orchestrator modules 2022-03-07T23:41:00.531 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:00.528784 I | op-osd: updating OSD 4 on PVC "smithi170-lv-2-data-0w6nq6" 2022-03-07T23:41:00.531 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:00.528804 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-2-data-0w6nq6" 2022-03-07T23:41:01.239 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:01.236832 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:41:01.333 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:01.330860 I | op-osd: updating OSD 5 on PVC "smithi166-lv-3-data-02vsxz" 2022-03-07T23:41:01.333 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:01.330896 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-3-data-02vsxz" 2022-03-07T23:41:02.040 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:02.038478 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:41:02.131 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:02.128940 I | op-osd: updating OSD 6 on PVC "smithi170-lv-4-data-0lsgm7" 2022-03-07T23:41:02.131 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:02.128969 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-4-data-0lsgm7" 2022-03-07T23:41:02.838 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:02.836702 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:41:02.932 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:02.929962 I | op-osd: updating OSD 7 on PVC "smithi166-lv-4-data-0cg6jn" 2022-03-07T23:41:02.932 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:02.929996 I | op-osd: OSD will have its main bluestore block on "smithi166-lv-4-data-0cg6jn" 2022-03-07T23:41:03.647 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:03.639653 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:41:03.731 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:03.729453 I | op-osd: updating OSD 8 on PVC "smithi170-lv-3-data-0rc4dr" 2022-03-07T23:41:03.732 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:03.729493 I | op-osd: OSD will have its main bluestore block on "smithi170-lv-3-data-0rc4dr" 2022-03-07T23:41:04.435 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:04.432674 I | op-osd: skipping osd checks for ok-to-stop 2022-03-07T23:41:04.547 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:04.545680 I | op-osd: updating OSD 9 on PVC "smithi078-lv-1-data-0mvnbx" 2022-03-07T23:41:04.548 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:04.545719 I | op-osd: OSD will have its main bluestore block on "smithi078-lv-1-data-0mvnbx" 2022-03-07T23:41:05.545 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:05.543102 I | op-osd: finished running OSDs in namespace "rook-ceph" 2022-03-07T23:41:05.545 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:05.543136 I | ceph-cluster-controller: done reconciling ceph cluster in namespace "rook-ceph" 2022-03-07T23:41:07.484 INFO:teuthology.orchestra.run.smithi078.stderr:+ '[' 11 '!=' 12 ']' 2022-03-07T23:41:07.485 INFO:teuthology.orchestra.run.smithi078.stderr:+ echo 'waiting for osd to come back up' 2022-03-07T23:41:07.485 INFO:teuthology.orchestra.run.smithi078.stdout:waiting for osd to come back up 2022-03-07T23:41:07.486 INFO:teuthology.orchestra.run.smithi078.stderr:++ toolbox ceph osd stat 2022-03-07T23:41:07.486 INFO:teuthology.orchestra.run.smithi078.stderr:++ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd stat 2022-03-07T23:41:07.486 INFO:teuthology.orchestra.run.smithi078.stderr:++ cut -f3 -d ' ' 2022-03-07T23:41:07.554 INFO:teuthology.orchestra.run.smithi078.stderr:Unable to use a TTY - input is not a terminal or the right kind of file 2022-03-07T23:41:08.027 INFO:teuthology.orchestra.run.smithi078.stderr:+ num_osd=12 2022-03-07T23:41:08.027 INFO:teuthology.orchestra.run.smithi078.stderr:+ sleep 30s 2022-03-07T23:41:38.029 INFO:teuthology.orchestra.run.smithi078.stderr:+ '[' 12 '!=' 12 ']' 2022-03-07T23:41:38.031 INFO:teuthology.run_tasks:Running task rook.shell... 2022-03-07T23:41:38.047 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch status 2022-03-07T23:41:38.764 INFO:teuthology.orchestra.run.smithi078.stdout:Backend: rook 2022-03-07T23:41:38.764 INFO:teuthology.orchestra.run.smithi078.stdout:Available: Yes 2022-03-07T23:41:38.797 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ps 2022-03-07T23:41:39.361 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID 2022-03-07T23:41:39.361 INFO:teuthology.orchestra.run.smithi078.stdout:crashcollector.crash smithi078 running (3m) 0s ago 3m - - 1c5288aef625 2022-03-07T23:41:39.362 INFO:teuthology.orchestra.run.smithi078.stdout:crashcollector.crash smithi166 running (5m) 0s ago 5m - - 1c5288aef625 2022-03-07T23:41:39.362 INFO:teuthology.orchestra.run.smithi078.stdout:crashcollector.crash smithi170 running (5m) 0s ago 5m - - 1c5288aef625 2022-03-07T23:41:39.362 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.a smithi078 running (5m) 0s ago 5m - - 1c5288aef625 2022-03-07T23:41:39.362 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi170 running (7m) 0s ago 7m - - 1c5288aef625 2022-03-07T23:41:39.363 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi166 running (7m) 0s ago 7m - - 1c5288aef625 2022-03-07T23:41:39.363 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi078 running (6m) 0s ago 6m - - 1c5288aef625 2022-03-07T23:41:39.363 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (65s) 0s ago 65s - - 1c5288aef625 2022-03-07T23:41:39.363 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi166 running (3m) 0s ago 3m - - 1c5288aef625 2022-03-07T23:41:39.363 INFO:teuthology.orchestra.run.smithi078.stdout:osd.10 smithi078 running (3m) 0s ago 3m - - 1c5288aef625 2022-03-07T23:41:39.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.11 smithi078 running (3m) 0s ago 3m - - 1c5288aef625 2022-03-07T23:41:39.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi170 running (3m) 0s ago 3m - - 1c5288aef625 2022-03-07T23:41:39.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi166 running (3m) 0s ago 3m - - 1c5288aef625 2022-03-07T23:41:39.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi170 running (3m) 0s ago 3m - - 1c5288aef625 2022-03-07T23:41:39.365 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi166 running (3m) 0s ago 3m - - 1c5288aef625 2022-03-07T23:41:39.365 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi170 running (3m) 0s ago 3m - - 1c5288aef625 2022-03-07T23:41:39.365 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi166 running (3m) 0s ago 3m - - 1c5288aef625 2022-03-07T23:41:39.365 INFO:teuthology.orchestra.run.smithi078.stdout:osd.8 smithi170 running (3m) 0s ago 3m - - 1c5288aef625 2022-03-07T23:41:39.365 INFO:teuthology.orchestra.run.smithi078.stdout:osd.9 smithi078 running (3m) 0s ago 3m - - 1c5288aef625 2022-03-07T23:41:39.383 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls 2022-03-07T23:41:40.099 INFO:teuthology.orchestra.run.smithi078.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-03-07T23:41:40.100 INFO:teuthology.orchestra.run.smithi078.stdout:crash 3/3 0s ago 5m * 2022-03-07T23:41:40.100 INFO:teuthology.orchestra.run.smithi078.stdout:mgr 1/1 0s ago 5m count:1 2022-03-07T23:41:40.100 INFO:teuthology.orchestra.run.smithi078.stdout:mon 3/3 0s ago 7m count:3 2022-03-07T23:41:40.101 INFO:teuthology.orchestra.run.smithi078.stdout:osd 24 0s ago 3m 2022-03-07T23:41:40.101 INFO:teuthology.orchestra.run.smithi078.stdout:osd.all-available-devices 0 0s ago - * 2022-03-07T23:41:40.124 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch host ls 2022-03-07T23:41:40.719 INFO:teuthology.orchestra.run.smithi078.stdout:HOST ADDR LABELS STATUS 2022-03-07T23:41:40.720 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078 172.21.15.78/smithi078 2022-03-07T23:41:40.721 INFO:teuthology.orchestra.run.smithi078.stdout:smithi166 172.21.15.166/smithi166 2022-03-07T23:41:40.721 INFO:teuthology.orchestra.run.smithi078.stdout:smithi170 172.21.15.170/smithi170 2022-03-07T23:41:40.721 INFO:teuthology.orchestra.run.smithi078.stdout:3 hosts in cluster 2022-03-07T23:41:40.741 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch device ls 2022-03-07T23:41:41.296 INFO:teuthology.orchestra.run.smithi078.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-03-07T23:41:41.297 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-07T23:41:41.297 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-07T23:41:41.297 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-07T23:41:41.298 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-07T23:41:41.298 INFO:teuthology.orchestra.run.smithi078.stdout:smithi166 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-07T23:41:41.298 INFO:teuthology.orchestra.run.smithi078.stdout:smithi166 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-07T23:41:41.298 INFO:teuthology.orchestra.run.smithi078.stdout:smithi166 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-07T23:41:41.298 INFO:teuthology.orchestra.run.smithi078.stdout:smithi166 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-07T23:41:41.299 INFO:teuthology.orchestra.run.smithi078.stdout:smithi170 /dev/vg_nvme/lv_1 unknown None 107G 0s ago 2022-03-07T23:41:41.299 INFO:teuthology.orchestra.run.smithi078.stdout:smithi170 /dev/vg_nvme/lv_2 unknown None 107G 0s ago 2022-03-07T23:41:41.299 INFO:teuthology.orchestra.run.smithi078.stdout:smithi170 /dev/vg_nvme/lv_3 unknown None 107G 0s ago 2022-03-07T23:41:41.299 INFO:teuthology.orchestra.run.smithi078.stdout:smithi170 /dev/vg_nvme/lv_4 unknown None 107G 0s ago 2022-03-07T23:41:41.315 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch apply rgw foo 2022-03-07T23:41:41.948 INFO:teuthology.orchestra.run.smithi078.stdout:Created 2022-03-07T23:41:41.949 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:41.947899 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "foo" 2022-03-07T23:41:41.970 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch apply mds foo 2022-03-07T23:41:42.029 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:42.027044 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:41:42.029 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:42.027114 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-07T23:41:42.511 INFO:teuthology.orchestra.run.smithi078.stdout:Created 2022-03-07T23:41:42.512 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:42.510724 I | ceph-spec: adding finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-07T23:41:42.533 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch apply rbd-mirror 2022-03-07T23:41:42.609 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:42.606992 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:41:42.609 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:42.607039 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-07T23:41:43.535 INFO:teuthology.orchestra.run.smithi078.stdout:Success 2022-03-07T23:41:43.552 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:43.550121 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:41:43.552 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:43.550202 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4... 2022-03-07T23:41:43.554 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch apply nfs foo --port 12777 2022-03-07T23:41:47.385 INFO:teuthology.orchestra.run.smithi078.stdout:Unable to create NFS daemon, check logs for more traceback 2022-03-07T23:41:47.386 INFO:teuthology.orchestra.run.smithi078.stdout:'RookOrchestrator' object has no attribute 'open_ioctx' 2022-03-07T23:41:47.410 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-03-07T23:41:47.425 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:41:48.125 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:41:48.126 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:33.000000Z", "last_refresh": "2022-03-07T23:41:47.956285Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:41:47.956285Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:41:47.956285Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:41:47.956285Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:41:47.956285Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:41:47.956285Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_name": "rbd-mirror", "service_type": "rbd-mirror", "status": {"last_refresh": "2022-03-07T23:41:47.956285Z", "running": 0, "size": 1}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:41:47.956285Z", "running": 0, "size": 1}}] 2022-03-07T23:41:48.153 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch rm mds.foo 2022-03-07T23:41:48.427 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:48.424919 I | ceph-spec: detected ceph image version: "17.0.0-11002 quincy" 2022-03-07T23:41:48.812 INFO:teuthology.orchestra.run.smithi078.stdout:Removed cephfilesystems/foo 2022-03-07T23:41:48.830 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch rm rbd-mirror 2022-03-07T23:41:49.985 INFO:teuthology.orchestra.run.smithi078.stdout:Removed cephrbdmirrors/default-rbd-mirror 2022-03-07T23:41:50.005 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch rm rgw.foo 2022-03-07T23:41:50.566 INFO:teuthology.orchestra.run.smithi078.stdout:Removed cephobjectstores/foo 2022-03-07T23:41:50.587 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:41:50.885 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:50.882196 I | ceph-spec: detected ceph image version: "17.0.0-11002 quincy" 2022-03-07T23:41:51.090 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:51.088835 I | ceph-file-controller: start running mdses for filesystem "foo" 2022-03-07T23:41:51.241 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:41:51.242 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:33.000000Z", "last_refresh": "2022-03-07T23:41:51.075577Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:41:51.075577Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:41:51.075577Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:41:51.075577Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:41:51.075577Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:41:51.075577Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:41:51.075577Z", "running": 0, "size": 1}}] 2022-03-07T23:41:51.478 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:51.476228 I | cephclient: getting or creating ceph auth key "mds.foo-a" 2022-03-07T23:41:52.256 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:52.253930 I | op-mds: setting mds config flags 2022-03-07T23:41:52.256 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:52.253956 I | op-config: setting "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-07T23:41:52.317 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:52.315280 I | ceph-object-controller: reconciling object store deployments 2022-03-07T23:41:52.409 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:52.407389 I | ceph-object-controller: ceph object store gateway service running at 10.104.143.190 2022-03-07T23:41:52.410 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:52.407430 I | ceph-object-controller: reconciling object store pools 2022-03-07T23:41:53.074 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:53.072362 W | ceph-object-controller: failed to adjust the PG count for rgw metadata pools. using the general default. failed to get config setting "rgw_rados_pool_pg_num_min" for user "mon.": exit status 2 2022-03-07T23:41:53.101 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:53.097627 I | op-config: successfully set "mds.foo-a"="mds_join_fs"="foo" option to the mon configuration database 2022-03-07T23:41:53.141 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:53.136973 I | cephclient: getting or creating ceph auth key "mds.foo-b" 2022-03-07T23:41:54.540 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:54.538094 I | op-mds: setting mds config flags 2022-03-07T23:41:54.540 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:54.538130 I | op-config: setting "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-07T23:41:55.926 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:55.924245 I | op-config: successfully set "mds.foo-b"="mds_join_fs"="foo" option to the mon configuration database 2022-03-07T23:41:56.620 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:56.617997 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi170": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T23:41:57.421 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:41:57.419718 I | ceph-file-controller: creating filesystem "foo" 2022-03-07T23:42:01.266 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:42:01.995 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:42:01.995 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:33.000000Z", "last_refresh": "2022-03-07T23:42:01.812027Z", "running": 5, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:42:01.812027Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:42:01.812027Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:42:01.812027Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:42:01.812027Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:42:01.812027Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:42:01.812027Z", "running": 0, "size": 1}}] 2022-03-07T23:42:02.622 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:02.620603 I | cephclient: reconciling replicated pool foo.rgw.log succeeded 2022-03-07T23:42:04.452 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:04.449689 I | cephclient: reconciling replicated pool foo.rgw.control succeeded 2022-03-07T23:42:04.454 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:04.451151 I | cephclient: reconciling replicated pool foo.rgw.buckets.non-ec succeeded 2022-03-07T23:42:04.455 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:04.452270 I | cephclient: reconciling replicated pool foo.rgw.meta succeeded 2022-03-07T23:42:04.460 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:04.458288 I | cephclient: reconciling replicated pool foo.rgw.buckets.index succeeded 2022-03-07T23:42:05.766 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:05.764710 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to create object store deployments: failed to create object pools: failed to create metadata pools: failed to create pool ".rgw.root": failed to tag pool ".rgw.root" for application "rook-ceph-rgw": failed to get application for pool ".rgw.root": failed to get current application for pool .rgw.root: exit status 2 2022-03-07T23:42:05.775 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:05.773812 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:42:05.892 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:05.890081 I | cephclient: reconciling replicated pool foo-metadata succeeded 2022-03-07T23:42:09.356 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:09.354415 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:42:09.345+0000 7fcff7aa3380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:42:09.357 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:42:09.345+0000 7fcff7aa3380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:42:09.357 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:42:09.345+0000 7fcff7aa3380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:42:09.357 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:42:09.387 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:09.385193 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:42:10.008 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:10.006353 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:42:09.969+0000 7ff5b9cb2380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:42:10.009 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:42:09.969+0000 7ff5b9cb2380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:42:10.009 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:42:09.969+0000 7ff5b9cb2380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:42:10.009 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:42:10.017 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:10.015696 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:42:10.694 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:10.692026 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:42:10.681+0000 7f86193d9380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:42:10.694 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:42:10.681+0000 7f86193d9380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:42:10.695 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:42:10.681+0000 7f86193d9380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:42:10.695 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:42:10.726 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:10.724498 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:42:11.135 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:11.132888 I | cephclient: reconciling replicated pool foo-data0 succeeded 2022-03-07T23:42:11.290 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:11.288013 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:42:11.277+0000 7f42dc272380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:42:11.290 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:42:11.277+0000 7f42dc272380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:42:11.291 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:42:11.277+0000 7f42dc272380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:42:11.291 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:42:11.327 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:11.325446 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:42:11.826 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:11.824691 I | cephclient: creating filesystem "foo" with metadata pool "foo-metadata" and data pools [foo-data0] 2022-03-07T23:42:11.900 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:11.898470 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:42:11.889+0000 7fe469405380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:42:11.900 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:42:11.889+0000 7fe469405380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:42:11.901 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:42:11.889+0000 7fe469405380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:42:11.901 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:42:11.964 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:11.962661 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:42:12.018 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:42:12.584 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:12.582055 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:42:12.573+0000 7f5c8c726380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:42:12.584 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:42:12.573+0000 7f5c8c726380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:42:12.585 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:42:12.573+0000 7f5c8c726380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:42:12.585 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:42:12.592 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:12.590532 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:42:12.752 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:42:12.752 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:42:12.588728Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:42:12.588728Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:42:12.588728Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:42:12.588728Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:42:12.588728Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:42:12.588728Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:42:12.588728Z", "running": 0, "size": 1}}] 2022-03-07T23:42:13.151 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:13.149145 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:42:13.141+0000 7f35d4344380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:42:13.151 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:42:13.141+0000 7f35d4344380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:42:13.152 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:42:13.141+0000 7f35d4344380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:42:13.152 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:42:13.192 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:13.190753 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:42:13.879 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:13.876716 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:42:13.869+0000 7f29655c8380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:42:13.879 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:42:13.869+0000 7f29655c8380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:42:13.879 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:42:13.869+0000 7f29655c8380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:42:13.879 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:42:13.892 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:13.890715 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:42:14.158 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:14.151394 I | ceph-file-controller: created filesystem "foo" on 1 data pool(s) and metadata pool "foo-metadata" 2022-03-07T23:42:14.159 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:14.151433 I | cephclient: setting allow_standby_replay for filesystem "foo" 2022-03-07T23:42:14.474 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:14.472086 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:42:14.465+0000 7f081812f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:42:14.474 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:42:14.465+0000 7f081812f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:42:14.474 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:42:14.465+0000 7f081812f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:42:14.475 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:42:15.166 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:15.164524 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:42:15.378 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:15.376419 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:42:15.383 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:15.382088 I | ceph-file-controller: CephFilesystem "rook-ceph/foo" can be deleted safely. deleting CephFilesystem "rook-ceph/foo" 2022-03-07T23:42:15.811 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:15.808670 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:42:15.801+0000 7f6a23361380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:42:15.811 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:42:15.801+0000 7f6a23361380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:42:15.812 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:42:15.801+0000 7f6a23361380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:42:15.812 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:42:16.158 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:16.156205 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-07T23:42:16.690 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:16.687742 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-07T23:42:16.690 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:16.687775 I | op-mds: successfully deleted mds config for "mds.foo-a" in mon configuration database 2022-03-07T23:42:16.691 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:16.687779 I | cephclient: deleting ceph auth "mds.foo-a" 2022-03-07T23:42:17.263 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:17.261207 I | op-mds: successfully deleted mds CephX key for "mds.foo-a" 2022-03-07T23:42:17.638 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:17.636965 I | op-config: deleting "mds_join_fs" option from the mon configuration database 2022-03-07T23:42:18.100 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:18.098710 I | op-config: successfully deleted "mds_join_fs" option from the mon configuration database 2022-03-07T23:42:18.101 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:18.098751 I | op-mds: successfully deleted mds config for "mds.foo-b" in mon configuration database 2022-03-07T23:42:18.101 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:18.098765 I | cephclient: deleting ceph auth "mds.foo-b" 2022-03-07T23:42:18.725 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:18.722543 I | op-mds: successfully deleted mds CephX key for "mds.foo-b" 2022-03-07T23:42:18.725 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:18.722580 I | ceph-file-controller: downing filesystem "foo" 2022-03-07T23:42:20.149 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:20.147381 I | ceph-file-controller: downed filesystem "foo" 2022-03-07T23:42:20.938 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:20.936266 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:42:21.521 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:21.518879 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:42:21.501+0000 7f4cbde6b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:42:21.521 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:42:21.501+0000 7f4cbde6b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:42:21.521 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:42:21.501+0000 7f4cbde6b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:42:21.522 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:42:22.776 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:42:23.987 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:42:23.988 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:42:23.789578Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "mds.foo", "service_type": "mds", "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:42:23.789578Z", "running": 0, "size": 2}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:42:23.789578Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:42:23.789578Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:42:23.789578Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:42:23.789578Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:42:23.789578Z", "running": 0, "size": 1}}] 2022-03-07T23:42:24.061 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:24.059739 I | cephclient: no images/snapshosts present in pool "foo-metadata" 2022-03-07T23:42:24.061 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:24.059760 I | cephclient: purging pool "foo-metadata" (id=8) 2022-03-07T23:42:26.741 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:26.739584 I | cephclient: purge completed for pool "foo-metadata" 2022-03-07T23:42:27.156 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:27.154178 I | cephclient: no images/snapshosts present in pool "foo-data0" 2022-03-07T23:42:27.156 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:27.154205 I | cephclient: purging pool "foo-data0" (id=10) 2022-03-07T23:42:29.664 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:29.662422 I | cephclient: purge completed for pool "foo-data0" 2022-03-07T23:42:29.665 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:29.662527 I | ceph-spec: removing finalizer "cephfilesystem.ceph.rook.io" on "foo" 2022-03-07T23:42:29.718 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:29.716259 I | ceph-spec: object "rook-ceph-mds-foo-a-keyring" matched on delete, reconciling 2022-03-07T23:42:29.724 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:29.722835 I | ceph-spec: object "rook-ceph-mds-foo-b-keyring" matched on delete, reconciling 2022-03-07T23:42:29.728 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:29.723800 I | ceph-spec: object "rook-ceph-mds-foo-b" matched on delete, reconciling 2022-03-07T23:42:29.728 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:29.723995 I | ceph-spec: object "rook-ceph-mds-foo-a" matched on delete, reconciling 2022-03-07T23:42:31.770 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:31.768689 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:42:32.353 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:32.351308 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi166": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T23:42:32.407 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:32.405080 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:42:32.397+0000 7f884b1b7380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:42:32.407 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:42:32.397+0000 7f884b1b7380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:42:32.407 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:42:32.397+0000 7f884b1b7380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:42:32.408 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:42:34.013 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:42:34.717 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:42:34.718 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:42:34.565781Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:42:34.565781Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:42:34.565781Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:42:34.565781Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:42:34.565781Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:42:34.565781Z", "running": 0, "size": 1}}] 2022-03-07T23:42:44.749 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:42:45.440 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:42:45.441 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:42:45.279500Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:42:45.279500Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:42:45.279500Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:42:45.279500Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:42:45.279500Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:42:45.279500Z", "running": 0, "size": 1}}] 2022-03-07T23:42:52.901 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:52.899222 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:42:53.544 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:42:53.542561 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:42:53.532+0000 7faf2f5cf380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:42:53.545 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:42:53.532+0000 7faf2f5cf380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:42:53.545 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:42:53.532+0000 7faf2f5cf380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:42:53.545 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:42:55.464 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:42:58.592 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:42:58.593 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:42:58.427169Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:42:58.427169Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:42:58.427169Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:42:58.427169Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:42:58.427169Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:42:58.427169Z", "running": 0, "size": 1}}] 2022-03-07T23:43:02.560 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:43:02.558157 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi170": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T23:43:02.769 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:43:02.767587 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi170": the object has been modified; please apply your changes to the latest version and try again 2022-03-07T23:43:08.617 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:43:09.319 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:43:09.319 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:43:09.133173Z", "running": 4, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:43:09.133173Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:43:09.133173Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:43:09.133173Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:43:09.133173Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:43:09.133173Z", "running": 0, "size": 1}}] 2022-03-07T23:43:19.353 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:43:20.075 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:43:20.076 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:43:19.890816Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:43:19.890816Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:43:19.890816Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:43:19.890816Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:43:19.890816Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:43:19.890816Z", "running": 0, "size": 1}}] 2022-03-07T23:43:30.100 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:43:30.789 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:43:30.789 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:43:30.628248Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:43:30.628248Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:43:30.628248Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:43:30.628248Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:43:30.628248Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:43:30.628248Z", "running": 0, "size": 1}}] 2022-03-07T23:43:34.513 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:43:34.511824 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:43:35.221 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:43:35.219212 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:43:35.191+0000 7f6b30e9e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:43:35.221 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:43:35.191+0000 7f6b30e9e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:43:35.222 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:43:35.191+0000 7f6b30e9e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:43:35.222 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:43:40.819 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:43:41.552 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:43:41.552 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:43:41.345306Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:43:41.345306Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:43:41.345306Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:43:41.345306Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:43:41.345306Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:43:41.345306Z", "running": 0, "size": 1}}] 2022-03-07T23:43:51.574 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:43:52.244 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:43:52.245 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:43:52.093482Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:43:52.093482Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:43:52.093482Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:43:52.093482Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:43:52.093482Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:43:52.093482Z", "running": 0, "size": 1}}] 2022-03-07T23:44:02.277 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:44:02.985 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:44:02.985 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:44:02.784366Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:44:02.784366Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:44:02.784366Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:44:02.784366Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:44:02.784366Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:44:02.784366Z", "running": 0, "size": 1}}] 2022-03-07T23:44:13.010 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:44:13.804 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:44:13.804 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:44:13.637153Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:44:13.637153Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:44:13.637153Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:44:13.637153Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:44:13.637153Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:44:13.637153Z", "running": 0, "size": 1}}] 2022-03-07T23:44:23.830 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:44:24.511 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:44:24.511 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:44:24.358966Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:44:24.358966Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:44:24.358966Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:44:24.358966Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:44:24.358966Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:44:24.358966Z", "running": 0, "size": 1}}] 2022-03-07T23:44:34.537 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:44:35.279 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:44:35.279 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:44:35.099225Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:44:35.099225Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:44:35.099225Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:44:35.099225Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:44:35.099225Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:44:35.099225Z", "running": 0, "size": 1}}] 2022-03-07T23:44:45.303 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:44:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:44:46.005 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:44:45.838114Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:44:45.838114Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:44:45.838114Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:44:45.838114Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:44:45.838114Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:44:45.838114Z", "running": 0, "size": 1}}] 2022-03-07T23:44:56.031 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:44:56.979 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:44:56.979 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:44:56.578352Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:44:56.578352Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:44:56.578352Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:44:56.578352Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:44:56.578352Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:44:56.578352Z", "running": 0, "size": 1}}] 2022-03-07T23:44:57.147 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:44:57.146003 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:44:57.826 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:44:57.824889 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:44:57.814+0000 7f3cec1ab380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:44:57.827 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:44:57.814+0000 7f3cec1ab380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:44:57.827 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:44:57.814+0000 7f3cec1ab380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:44:57.827 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:45:07.009 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:45:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:45:07.716 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:45:07.561548Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:45:07.561548Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:45:07.561548Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:45:07.561548Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:45:07.561548Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:45:07.561548Z", "running": 0, "size": 1}}] 2022-03-07T23:45:17.738 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:45:18.427 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:45:18.427 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:45:18.262839Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:45:18.262839Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:45:18.262839Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:45:18.262839Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:45:18.262839Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:45:18.262839Z", "running": 0, "size": 1}}] 2022-03-07T23:45:28.452 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:45:29.140 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:45:29.140 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:45:28.980031Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:45:28.980031Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:45:28.980031Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:45:28.980031Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:45:28.980031Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:45:28.980031Z", "running": 0, "size": 1}}] 2022-03-07T23:45:39.187 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:45:39.941 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:45:39.941 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:45:39.736662Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:45:39.736662Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:45:39.736662Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:45:39.736662Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:45:39.736662Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:45:39.736662Z", "running": 0, "size": 1}}] 2022-03-07T23:45:49.966 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:45:50.657 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:45:50.657 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:45:50.489187Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:45:50.489187Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:45:50.489187Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:45:50.489187Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:45:50.489187Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:45:50.489187Z", "running": 0, "size": 1}}] 2022-03-07T23:46:00.683 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:46:01.349 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:46:01.443 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:46:01.185446Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:46:01.185446Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:46:01.185446Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:46:01.185446Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:46:01.185446Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:46:01.185446Z", "running": 0, "size": 1}}] 2022-03-07T23:46:11.446 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:46:12.184 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:46:12.184 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:46:11.983116Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:46:11.983116Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:46:11.983116Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:46:11.983116Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:46:11.983116Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:46:11.983116Z", "running": 0, "size": 1}}] 2022-03-07T23:46:22.210 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:46:22.992 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:46:22.993 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:46:22.782358Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:46:22.782358Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:46:22.782358Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:46:22.782358Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:46:22.782358Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:46:22.782358Z", "running": 0, "size": 1}}] 2022-03-07T23:46:33.019 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:46:33.771 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:46:33.772 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:46:33.582502Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:46:33.582502Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:46:33.582502Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:46:33.582502Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:46:33.582502Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:46:33.582502Z", "running": 0, "size": 1}}] 2022-03-07T23:46:43.799 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:46:44.480 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:46:44.480 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:46:44.328976Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:46:44.328976Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:46:44.328976Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:46:44.328976Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:46:44.328976Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:46:44.328976Z", "running": 0, "size": 1}}] 2022-03-07T23:46:54.509 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:46:55.182 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:46:55.183 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:46:55.018834Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:46:55.018834Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:46:55.018834Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:46:55.018834Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:46:55.018834Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:46:55.018834Z", "running": 0, "size": 1}}] 2022-03-07T23:47:05.217 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:47:05.904 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:47:05.904 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:47:05.752490Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:47:05.752490Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:47:05.752490Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:47:05.752490Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:47:05.752490Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:47:05.752490Z", "running": 0, "size": 1}}] 2022-03-07T23:47:15.932 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:47:16.648 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:47:16.821 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:47:16.462089Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:47:16.462089Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:47:16.462089Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:47:16.462089Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:47:16.462089Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:47:16.462089Z", "running": 0, "size": 1}}] 2022-03-07T23:47:26.824 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:47:27.589 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:47:27.590 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:47:27.436628Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:47:27.436628Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:47:27.436628Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:47:27.436628Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:47:27.436628Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:47:27.436628Z", "running": 0, "size": 1}}] 2022-03-07T23:47:37.614 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:47:38.330 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:47:38.331 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:47:38.133867Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:47:38.133867Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:47:38.133867Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:47:38.133867Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:47:38.133867Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:47:38.133867Z", "running": 0, "size": 1}}] 2022-03-07T23:47:41.675 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:47:41.673116 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:47:42.448 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:47:42.445871 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:47:42.431+0000 7fb8d63be380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:47:42.448 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:47:42.431+0000 7fb8d63be380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:47:42.448 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:47:42.431+0000 7fb8d63be380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:47:42.449 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:47:48.358 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:47:49.151 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:47:49.152 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:47:48.946644Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:47:48.946644Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:47:48.946644Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:47:48.946644Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:47:48.946644Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:47:48.946644Z", "running": 0, "size": 1}}] 2022-03-07T23:47:59.179 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:47:59.871 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:47:59.871 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:47:59.695997Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:47:59.695997Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:47:59.695997Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:47:59.695997Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:47:59.695997Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:47:59.695997Z", "running": 0, "size": 1}}] 2022-03-07T23:48:09.904 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:48:10.585 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:48:10.586 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:48:10.419547Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:48:10.419547Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:48:10.419547Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:48:10.419547Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:48:10.419547Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:48:10.419547Z", "running": 0, "size": 1}}] 2022-03-07T23:48:20.614 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:48:21.287 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:48:21.287 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:48:21.132225Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:48:21.132225Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:48:21.132225Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:48:21.132225Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:48:21.132225Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:48:21.132225Z", "running": 0, "size": 1}}] 2022-03-07T23:48:31.315 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:48:31.996 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:48:31.996 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:48:31.811425Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:48:31.811425Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:48:31.811425Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:48:31.811425Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:48:31.811425Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:48:31.811425Z", "running": 0, "size": 1}}] 2022-03-07T23:48:42.022 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:48:42.802 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:48:42.803 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:48:42.588243Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:48:42.588243Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:48:42.588243Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:48:42.588243Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:48:42.588243Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:48:42.588243Z", "running": 0, "size": 1}}] 2022-03-07T23:48:52.826 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:48:53.493 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:48:53.494 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:48:53.324349Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:48:53.324349Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:48:53.324349Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:48:53.324349Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:48:53.324349Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:48:53.324349Z", "running": 0, "size": 1}}] 2022-03-07T23:49:03.559 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:49:04.312 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:49:04.313 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:49:04.111747Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:49:04.111747Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:49:04.111747Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:49:04.111747Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:49:04.111747Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:49:04.111747Z", "running": 0, "size": 1}}] 2022-03-07T23:49:14.339 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:49:14.993 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:49:14.993 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:49:14.827839Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:49:14.827839Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:49:14.827839Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:49:14.827839Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:49:14.827839Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:49:14.827839Z", "running": 0, "size": 1}}] 2022-03-07T23:49:25.013 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:49:25.730 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:49:25.730 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:49:25.553208Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:49:25.553208Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:49:25.553208Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:49:25.553208Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:49:25.553208Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:49:25.553208Z", "running": 0, "size": 1}}] 2022-03-07T23:49:35.758 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:49:36.474 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:49:36.475 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:49:36.318456Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:49:36.318456Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:49:36.318456Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:49:36.318456Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:49:36.318456Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:49:36.318456Z", "running": 0, "size": 1}}] 2022-03-07T23:49:46.505 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:49:47.194 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:49:47.195 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:49:47.042309Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:49:47.042309Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:49:47.042309Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:49:47.042309Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:49:47.042309Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:49:47.042309Z", "running": 0, "size": 1}}] 2022-03-07T23:49:57.220 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:49:57.915 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:49:57.915 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:49:57.753748Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:49:57.753748Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:49:57.753748Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:49:57.753748Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:49:57.753748Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:49:57.753748Z", "running": 0, "size": 1}}] 2022-03-07T23:50:07.940 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:50:08.632 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:50:08.632 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:50:08.460515Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:50:08.460515Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:50:08.460515Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:50:08.460515Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:50:08.460515Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:50:08.460515Z", "running": 0, "size": 1}}] 2022-03-07T23:50:18.658 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:50:19.343 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:50:19.344 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:50:19.186891Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:50:19.186891Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:50:19.186891Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:50:19.186891Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:50:19.186891Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:50:19.186891Z", "running": 0, "size": 1}}] 2022-03-07T23:50:29.372 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:50:30.069 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:50:30.069 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:50:29.927741Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:50:29.927741Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:50:29.927741Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:50:29.927741Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:50:29.927741Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:50:29.927741Z", "running": 0, "size": 1}}] 2022-03-07T23:50:40.095 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:50:40.776 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:50:40.776 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:50:40.597979Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:50:40.597979Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:50:40.597979Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:50:40.597979Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:50:40.597979Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:50:40.597979Z", "running": 0, "size": 1}}] 2022-03-07T23:50:50.800 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:50:51.487 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:50:51.488 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:50:51.325453Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:50:51.325453Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:50:51.325453Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:50:51.325453Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:50:51.325453Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:50:51.325453Z", "running": 0, "size": 1}}] 2022-03-07T23:51:01.513 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:51:02.226 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:51:02.226 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:51:02.071832Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:51:02.071832Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:51:02.071832Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:51:02.071832Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:51:02.071832Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:51:02.071832Z", "running": 0, "size": 1}}] 2022-03-07T23:51:12.252 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:51:12.918 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:51:12.918 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:51:12.761388Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:51:12.761388Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:51:12.761388Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:51:12.761388Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:51:12.761388Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:51:12.761388Z", "running": 0, "size": 1}}] 2022-03-07T23:51:22.944 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:51:23.641 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:51:23.641 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:51:23.471063Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:51:23.471063Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:51:23.471063Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:51:23.471063Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:51:23.471063Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:51:23.471063Z", "running": 0, "size": 1}}] 2022-03-07T23:51:33.672 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:51:34.391 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:51:34.391 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:51:34.206931Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:51:34.206931Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:51:34.206931Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:51:34.206931Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:51:34.206931Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:51:34.206931Z", "running": 0, "size": 1}}] 2022-03-07T23:51:44.412 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:51:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:51:45.096 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:51:44.932038Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:51:44.932038Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:51:44.932038Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:51:44.932038Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:51:44.932038Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:51:44.932038Z", "running": 0, "size": 1}}] 2022-03-07T23:51:55.121 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:51:55.795 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:51:55.796 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:51:55.639827Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:51:55.639827Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:51:55.639827Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:51:55.639827Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:51:55.639827Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:51:55.639827Z", "running": 0, "size": 1}}] 2022-03-07T23:52:05.831 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:52:06.520 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:52:06.520 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:52:06.362213Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:52:06.362213Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:52:06.362213Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:52:06.362213Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:52:06.362213Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:52:06.362213Z", "running": 0, "size": 1}}] 2022-03-07T23:52:16.545 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:52:17.227 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:52:17.228 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:52:17.074822Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:52:17.074822Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:52:17.074822Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:52:17.074822Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:52:17.074822Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:52:17.074822Z", "running": 0, "size": 1}}] 2022-03-07T23:52:27.251 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:52:28.006 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:52:28.007 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:52:27.790373Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:52:27.790373Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:52:27.790373Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:52:27.790373Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:52:27.790373Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:52:27.790373Z", "running": 0, "size": 1}}] 2022-03-07T23:52:38.034 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:52:38.719 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:52:38.719 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:52:38.571215Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:52:38.571215Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:52:38.571215Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:52:38.571215Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:52:38.571215Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:52:38.571215Z", "running": 0, "size": 1}}] 2022-03-07T23:52:48.746 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:52:49.420 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:52:49.421 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:52:49.264919Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:52:49.264919Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:52:49.264919Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:52:49.264919Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:52:49.264919Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:52:49.264919Z", "running": 0, "size": 1}}] 2022-03-07T23:52:59.454 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:53:02.863 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:53:02.864 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:53:02.601429Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:53:02.601429Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:53:02.601429Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:53:02.601429Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:53:02.601429Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:53:02.601429Z", "running": 0, "size": 1}}] 2022-03-07T23:53:10.134 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:53:10.133595 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:53:10.774 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:53:10.773235 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:53:10.761+0000 7f3908078380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:53:10.775 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:53:10.761+0000 7f3908078380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:53:10.775 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:53:10.761+0000 7f3908078380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:53:10.775 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:53:12.884 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:53:13.592 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:53:13.593 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:53:13.422448Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:53:13.422448Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:53:13.422448Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:53:13.422448Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:53:13.422448Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:53:13.422448Z", "running": 0, "size": 1}}] 2022-03-07T23:53:23.617 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:53:24.289 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:53:24.289 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:53:24.136040Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:53:24.136040Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:53:24.136040Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:53:24.136040Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:53:24.136040Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:53:24.136040Z", "running": 0, "size": 1}}] 2022-03-07T23:53:34.316 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:53:35.152 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:53:35.153 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:53:34.929161Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:53:34.929161Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:53:34.929161Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:53:34.929161Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:53:34.929161Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:53:34.929161Z", "running": 0, "size": 1}}] 2022-03-07T23:53:45.196 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:53:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:53:45.929 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:53:45.749557Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:53:45.749557Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:53:45.749557Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:53:45.749557Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:53:45.749557Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:53:45.749557Z", "running": 0, "size": 1}}] 2022-03-07T23:53:55.964 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:53:56.644 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:53:56.645 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:53:56.467183Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:53:56.467183Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:53:56.467183Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:53:56.467183Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:53:56.467183Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:53:56.467183Z", "running": 0, "size": 1}}] 2022-03-07T23:54:06.670 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:54:07.422 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:54:07.422 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:54:07.211098Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:54:07.211098Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:54:07.211098Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:54:07.211098Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:54:07.211098Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:54:07.211098Z", "running": 0, "size": 1}}] 2022-03-07T23:54:17.450 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:54:18.172 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:54:18.173 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:54:17.978269Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:54:17.978269Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:54:17.978269Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:54:17.978269Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:54:17.978269Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:54:17.978269Z", "running": 0, "size": 1}}] 2022-03-07T23:54:28.200 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:54:28.929 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:54:28.929 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:54:28.760261Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:54:28.760261Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:54:28.760261Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:54:28.760261Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:54:28.760261Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:54:28.760261Z", "running": 0, "size": 1}}] 2022-03-07T23:54:38.955 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:54:39.643 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:54:39.644 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:54:39.478935Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:54:39.478935Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:54:39.478935Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:54:39.478935Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:54:39.478935Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:54:39.478935Z", "running": 0, "size": 1}}] 2022-03-07T23:54:49.670 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:54:50.427 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:54:50.428 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:54:50.216631Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:54:50.216631Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:54:50.216631Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:54:50.216631Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:54:50.216631Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:54:50.216631Z", "running": 0, "size": 1}}] 2022-03-07T23:55:00.454 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:55:01.132 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:55:01.132 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:55:00.981413Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:55:00.981413Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:55:00.981413Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:55:00.981413Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:55:00.981413Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:55:00.981413Z", "running": 0, "size": 1}}] 2022-03-07T23:55:11.156 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:55:11.857 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:55:11.857 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:55:11.678360Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:55:11.678360Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:55:11.678360Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:55:11.678360Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:55:11.678360Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:55:11.678360Z", "running": 0, "size": 1}}] 2022-03-07T23:55:21.886 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:55:22.977 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:55:22.977 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:55:22.471050Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:55:22.471050Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:55:22.471050Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:55:22.471050Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:55:22.471050Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:55:22.471050Z", "running": 0, "size": 1}}] 2022-03-07T23:55:33.016 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:55:33.741 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:55:33.742 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:55:33.550077Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:55:33.550077Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:55:33.550077Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:55:33.550077Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:55:33.550077Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:55:33.550077Z", "running": 0, "size": 1}}] 2022-03-07T23:55:43.767 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:55:44.448 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:55:44.448 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:55:44.304353Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:55:44.304353Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:55:44.304353Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:55:44.304353Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:55:44.304353Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:55:44.304353Z", "running": 0, "size": 1}}] 2022-03-07T23:55:54.473 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:55:55.187 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:55:55.187 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:55:55.014657Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:55:55.014657Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:55:55.014657Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:55:55.014657Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:55:55.014657Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:55:55.014657Z", "running": 0, "size": 1}}] 2022-03-07T23:56:05.218 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:56:05.899 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:56:05.899 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:56:05.742993Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:56:05.742993Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:56:05.742993Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:56:05.742993Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:56:05.742993Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:56:05.742993Z", "running": 0, "size": 1}}] 2022-03-07T23:56:15.926 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:56:16.584 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:56:16.585 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:56:16.431314Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:56:16.431314Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:56:16.431314Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:56:16.431314Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:56:16.431314Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:56:16.431314Z", "running": 0, "size": 1}}] 2022-03-07T23:56:26.608 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:56:27.390 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:56:27.391 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:56:27.212705Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:56:27.212705Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:56:27.212705Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:56:27.212705Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:56:27.212705Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:56:27.212705Z", "running": 0, "size": 1}}] 2022-03-07T23:56:37.416 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:56:38.090 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:56:38.090 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:56:37.937759Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:56:37.937759Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:56:37.937759Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:56:37.937759Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:56:37.937759Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:56:37.937759Z", "running": 0, "size": 1}}] 2022-03-07T23:56:43.581 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:56:43.580157 E | ceph-rbd-mirror-controller: failed to reconcile failed to detect running and desired ceph version: failed to detect ceph image version: failed to complete ceph version job: failed to run CmdReporter ceph-rbd-mirror-controller-detect-version successfully. failed waiting for results ConfigMap ceph-rbd-mirror-controller-detect-version. timed out waiting for results ConfigMap 2022-03-07T23:56:48.116 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:56:48.871 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:56:48.872 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:56:48.653196Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:56:48.653196Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:56:48.653196Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:56:48.653196Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:56:48.653196Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:56:48.653196Z", "running": 0, "size": 1}}] 2022-03-07T23:56:58.896 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:56:59.619 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:56:59.620 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:56:59.458147Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:56:59.458147Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:56:59.458147Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:56:59.458147Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:56:59.458147Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:56:59.458147Z", "running": 0, "size": 1}}] 2022-03-07T23:57:09.646 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:57:10.391 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:57:10.392 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:57:10.233033Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:57:10.233033Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:57:10.233033Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:57:10.233033Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:57:10.233033Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:57:10.233033Z", "running": 0, "size": 1}}] 2022-03-07T23:57:20.418 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:57:21.119 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:57:21.119 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:57:20.937275Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:57:20.937275Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:57:20.937275Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:57:20.937275Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:57:20.937275Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:57:20.937275Z", "running": 0, "size": 1}}] 2022-03-07T23:57:31.142 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:57:31.813 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:57:31.813 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:57:31.657309Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:57:31.657309Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:57:31.657309Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:57:31.657309Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:57:31.657309Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:57:31.657309Z", "running": 0, "size": 1}}] 2022-03-07T23:57:41.840 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:57:42.541 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:57:42.541 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:57:42.377094Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:57:42.377094Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:57:42.377094Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:57:42.377094Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:57:42.377094Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:57:42.377094Z", "running": 0, "size": 1}}] 2022-03-07T23:57:52.568 DEBUG:teuthology.orchestra.run.smithi078:> kubectl -n rook-ceph exec rook-ceph-tools-d6d7c985c-jnk6p -- ceph orch ls -f json 2022-03-07T23:57:53.322 INFO:teuthology.orchestra.run.smithi078.stdout: 2022-03-07T23:57:53.323 INFO:teuthology.orchestra.run.smithi078.stdout:[{"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:55.000000Z", "last_refresh": "2022-03-07T23:57:53.158651Z", "running": 3, "size": 3}}, {"placement": {"count": 1}, "service_name": "mgr", "service_type": "mgr", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:36:31.000000Z", "last_refresh": "2022-03-07T23:57:53.158651Z", "running": 1, "size": 1}}, {"placement": {"count": 3}, "service_name": "mon", "service_type": "mon", "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:34:03.000000Z", "last_refresh": "2022-03-07T23:57:53.158651Z", "running": 3, "size": 3}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "1c5288aef6256a889df892d75ab463939ebf6a7dae8971c4bd12eb21c485b2fb", "container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "created": "2022-03-07T23:37:59.000000Z", "last_refresh": "2022-03-07T23:57:53.158651Z", "running": 24, "size": 12}, "unmanaged": true}, {"placement": {"host_pattern": "*"}, "service_id": "all-available-devices", "service_name": "osd.all-available-devices", "service_type": "osd", "spec": {"data_devices": {"all": true}, "filter_logic": "AND", "objectstore": "bluestore"}, "status": {"last_refresh": "2022-03-07T23:57:53.158651Z", "running": 0, "size": 0}}, {"placement": {"count": 1}, "service_id": "foo", "service_name": "rgw.foo", "service_type": "rgw", "spec": {"rgw_frontend_port": 80}, "status": {"container_image_name": "quay.ceph.io/ceph-ci/ceph:94cb3430b74fb099b8927e3e6ad50dfd969173f4", "last_refresh": "2022-03-07T23:57:53.158651Z", "running": 0, "size": 1}}] 2022-03-07T23:57:53.349 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-07T23:57:53.349 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 530, in ceph_config_keyring yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-07T23:57:53.350 INFO:tasks.rook:Cleaning up config and client.admin keyring 2022-03-07T23:57:53.350 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-07T23:57:53.365 DEBUG:teuthology.orchestra.run.smithi166:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-07T23:57:53.383 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2022-03-07T23:57:53.399 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 478, in rook_post_config yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-07T23:57:53.400 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 442, in rook_toolbox yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-07T23:57:53.410 DEBUG:teuthology.orchestra.remote:smithi078:rook/deploy/examples/operator.yaml is 20KB 2022-03-07T23:57:53.424 DEBUG:teuthology.orchestra.run.smithi078:> kubectl delete -f rook/deploy/examples/toolbox.yaml 2022-03-07T23:57:53.492 INFO:teuthology.orchestra.run.smithi078.stdout:deployment.apps "rook-ceph-tools" deleted 2022-03-07T23:57:53.503 ERROR:tasks.rook:'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 379, in rook_cluster yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_94cb3430b74fb099b8927e3e6ad50dfd969173f4/qa/tasks/rook.py", line 669, in task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'waiting for service removal' reached maximum tries (90) after waiting for 900 seconds 2022-03-07T23:57:53.504 DEBUG:teuthology.orchestra.run.smithi078:> kubectl delete -f cluster.yaml 2022-03-07T23:57:53.593 INFO:teuthology.orchestra.run.smithi078.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-03-07T23:57:53.600 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.598851 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-03-07T23:57:53.600 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.599002 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-03-07T23:57:53.603 INFO:tasks.rook.operator.smithi078.stdout:I0307 23:57:53.599509 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"={} 2022-03-07T23:57:53.603 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.600033 I | operator: successfully started the controller-runtime manager 2022-03-07T23:57:53.616 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.615858 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-03-07T23:57:53.617 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.615877 I | operator: watching all namespaces for Ceph CRs 2022-03-07T23:57:53.617 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.615904 I | operator: setting up schemes 2022-03-07T23:57:53.620 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.618182 I | operator: setting up the controller-runtime manager 2022-03-07T23:57:53.822 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.821141 I | operator: looking for admission webhook secret "rook-ceph-admission-controller" 2022-03-07T23:57:53.881 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.879625 I | operator: admission webhook secret "rook-ceph-admission-controller" not found. proceeding without the admission controller 2022-03-07T23:57:53.882 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.879671 I | ceph-cluster-controller: successfully started 2022-03-07T23:57:53.882 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.879731 I | ceph-cluster-controller: enabling hotplug orchestration 2022-03-07T23:57:53.882 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.879755 I | ceph-crashcollector-controller: successfully started 2022-03-07T23:57:53.882 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.879781 I | ceph-block-pool-controller: successfully started 2022-03-07T23:57:53.883 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.879816 I | ceph-object-store-user-controller: successfully started 2022-03-07T23:57:53.883 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.879858 I | ceph-object-realm-controller: successfully started 2022-03-07T23:57:53.883 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.879889 I | ceph-object-zonegroup-controller: successfully started 2022-03-07T23:57:53.883 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.879916 I | ceph-object-zone-controller: successfully started 2022-03-07T23:57:53.883 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.880117 I | ceph-object-controller: successfully started 2022-03-07T23:57:53.884 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.880170 I | ceph-file-controller: successfully started 2022-03-07T23:57:53.884 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.880225 I | ceph-nfs-controller: successfully started 2022-03-07T23:57:53.884 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.880293 I | ceph-rbd-mirror-controller: successfully started 2022-03-07T23:57:53.885 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.880348 I | ceph-client-controller: successfully started 2022-03-07T23:57:53.885 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.880387 I | ceph-filesystem-mirror-controller: successfully started 2022-03-07T23:57:53.886 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.880440 I | operator: rook-ceph-operator-config-controller successfully started 2022-03-07T23:57:53.887 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.880479 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-03-07T23:57:53.887 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.880511 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-03-07T23:57:53.888 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.880546 I | ceph-bucket-topic: successfully started 2022-03-07T23:57:53.888 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.880574 I | ceph-bucket-notification: successfully started 2022-03-07T23:57:53.888 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.880604 I | ceph-bucket-notification: successfully started 2022-03-07T23:57:53.888 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.880629 I | ceph-fs-subvolumegroup-controller: successfully started 2022-03-07T23:57:53.888 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.882142 I | operator: starting the controller-runtime manager 2022-03-07T23:57:53.988 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.987361 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-03-07T23:57:53.988 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.987419 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-03-07T23:57:53.989 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.987432 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-03-07T23:57:53.992 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.990736 I | operator: rook-ceph-operator-config-controller done reconciling 2022-03-07T23:57:53.997 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:53.992612 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:57:54.069 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:54.066247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-07T23:57:54.076 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:54.075901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-07T23:57:54.684 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:54.683153 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:57:54.672+0000 7fb46dd7e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:57:54.684 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:57:54.672+0000 7fb46dd7e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:57:54.685 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:57:54.672+0000 7fb46dd7e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:57:54.685 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:57:54.700 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:54.700041 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:57:55.436 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:55.435340 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:57:55.424+0000 7fc877c63380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:57:55.436 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:57:55.424+0000 7fc877c63380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:57:55.436 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:57:55.424+0000 7fc877c63380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:57:55.437 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:57:55.450 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:55.450015 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:57:56.094 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:56.093079 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:57:56.079+0000 7fefd5dc2380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:57:56.094 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:57:56.079+0000 7fefd5dc2380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:57:56.095 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:57:56.079+0000 7fefd5dc2380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:57:56.095 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:57:56.120 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:56.119773 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:57:56.774 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:56.773190 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:57:56.759+0000 7ff8e9ab3380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:57:56.774 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:57:56.759+0000 7ff8e9ab3380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:57:56.775 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:57:56.759+0000 7ff8e9ab3380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:57:56.775 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:57:56.822 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:56.822348 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:57:57.543 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:57.542022 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:57:57.527+0000 7fc1afb1e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:57:57.543 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:57:57.527+0000 7fc1afb1e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:57:57.543 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:57:57.527+0000 7fc1afb1e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:57:57.544 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:57:57.630 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:57.629626 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:57:58.262 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:58.261585 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:57:58.247+0000 7f47381a3380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:57:58.263 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:57:58.247+0000 7f47381a3380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:57:58.263 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:57:58.247+0000 7f47381a3380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:57:58.263 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:57:58.429 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:58.429128 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:57:59.077 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:59.076667 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:57:59.063+0000 7f62150de380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:57:59.077 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:57:59.063+0000 7f62150de380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:57:59.078 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:57:59.063+0000 7f62150de380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:57:59.078 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:57:59.405 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:57:59.404453 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:58:00.373 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:58:00.372605 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:58:00.359+0000 7f160d067380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:58:00.374 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:58:00.359+0000 7f160d067380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:58:00.374 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:58:00.359+0000 7f160d067380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:58:00.374 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:58:01.019 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:58:01.018601 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:58:01.659 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:58:01.658404 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:58:01.643+0000 7fe78d42b380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:58:01.660 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:58:01.643+0000 7fe78d42b380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:58:01.660 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:58:01.643+0000 7fe78d42b380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:58:01.661 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:58:02.946 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:58:02.945960 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:58:03.643 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:58:03.642655 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:58:03.631+0000 7f268eb6c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:58:03.644 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:58:03.631+0000 7f268eb6c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:58:03.644 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:58:03.631+0000 7f268eb6c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:58:03.644 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:58:04.173 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:58:04.172947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-07T23:58:04.187 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:58:04.187213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-07T23:58:06.212 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:58:06.210637 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:58:06.864 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:58:06.863271 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:58:06.851+0000 7fa783a33380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:58:06.864 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:58:06.851+0000 7fa783a33380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:58:06.865 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:58:06.851+0000 7fa783a33380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:58:06.865 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:58:11.993 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:58:11.992371 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:58:12.676 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:58:12.675603 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:58:12.631+0000 7fa4a3cca380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:58:12.677 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:58:12.631+0000 7fa4a3cca380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:58:12.677 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:58:12.631+0000 7fa4a3cca380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:58:12.677 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:58:14.303 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:58:14.302588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-07T23:58:14.315 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:58:14.314761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-07T23:58:22.922 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:58:22.921312 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:58:23.586 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:58:23.585467 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:58:23.571+0000 7fed9ff75380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:58:23.586 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:58:23.571+0000 7fed9ff75380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:58:23.587 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:58:23.571+0000 7fed9ff75380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:58:23.587 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:58:24.420 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:58:24.419747 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-07T23:58:24.430 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:58:24.429548 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-07T23:58:34.530 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:58:34.529240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-07T23:58:34.542 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:58:34.542023 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-07T23:58:44.074 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:58:44.072739 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:58:44.610 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:58:44.610189 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-07T23:58:44.625 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:58:44.623802 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-07T23:58:44.729 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:58:44.728322 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:58:44.719+0000 7f68eeb4e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:58:44.729 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:58:44.719+0000 7f68eeb4e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:58:44.730 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:58:44.719+0000 7f68eeb4e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:58:44.730 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:58:54.697 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:58:54.696698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-07T23:58:54.751 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:58:54.747508 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-07T23:59:04.870 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:59:04.869379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-07T23:59:04.880 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:59:04.879364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-07T23:59:14.952 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:59:14.951684 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-07T23:59:15.136 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:59:14.963222 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-07T23:59:25.081 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:59:25.079957 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-07T23:59:25.093 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:59:25.092635 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-07T23:59:25.696 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:59:25.695945 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-07T23:59:26.350 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:59:26.349590 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-07T23:59:26.338+0000 7fd91ee40380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-07T23:59:26.351 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:59:26.338+0000 7fd91ee40380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-07T23:59:26.351 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07T23:59:26.338+0000 7fd91ee40380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-07T23:59:26.351 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-07T23:59:35.156 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:59:35.155811 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-07T23:59:35.169 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:59:35.168575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-07T23:59:45.461 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:59:45.459534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-07T23:59:45.474 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:59:45.474173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-07T23:59:55.613 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:59:55.613227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-07T23:59:55.633 INFO:tasks.rook.operator.smithi078.stdout:2022-03-07 23:59:55.630436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:00:05.700 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:00:05.700138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:00:05.711 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:00:05.710636 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:00:15.777 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:00:15.776994 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:00:15.791 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:00:15.790805 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:00:25.856 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:00:25.855303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:00:25.868 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:00:25.867734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:00:35.981 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:00:35.980450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:00:35.995 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:00:35.991959 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:00:46.095 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:00:46.094476 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:00:46.107 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:00:46.107419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:00:48.277 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:00:48.276836 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-08T00:00:48.934 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:00:48.933006 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T00:00:48.920+0000 7fb8c169e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T00:00:48.934 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T00:00:48.920+0000 7fb8c169e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T00:00:48.935 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T00:00:48.920+0000 7fb8c169e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T00:00:48.935 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-08T00:00:56.205 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:00:56.204655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:00:56.219 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:00:56.219119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:01:06.326 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:01:06.326047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:01:06.337 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:01:06.337226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:01:16.433 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:01:16.432953 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:01:16.469 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:01:16.468864 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:01:26.563 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:01:26.560867 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:01:26.573 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:01:26.573297 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:01:36.665 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:01:36.664484 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:01:36.676 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:01:36.675631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:01:46.785 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:01:46.784271 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:01:46.797 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:01:46.796949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:01:56.892 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:01:56.891708 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:01:56.903 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:01:56.903511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:02:06.979 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:02:06.978633 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:02:06.990 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:02:06.990092 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:02:17.066 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:02:17.066308 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:02:17.078 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:02:17.078173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:02:27.179 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:02:27.179089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:02:27.192 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:02:27.192410 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:02:37.298 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:02:37.297963 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:02:37.310 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:02:37.310531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:02:47.435 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:02:47.434585 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:02:47.448 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:02:47.447868 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:02:57.556 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:02:57.555604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:02:57.571 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:02:57.571313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:03:07.684 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:03:07.683740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:03:07.697 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:03:07.696607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:03:17.771 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:03:17.771477 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:03:17.783 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:03:17.783003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:03:27.885 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:03:27.884655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:03:27.905 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:03:27.904904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:03:32.780 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:03:32.780286 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-08T00:03:33.426 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:03:33.426101 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T00:03:33.413+0000 7fed0b40e380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T00:03:33.427 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T00:03:33.413+0000 7fed0b40e380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T00:03:33.427 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T00:03:33.413+0000 7fed0b40e380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T00:03:33.427 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-08T00:03:38.027 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:03:38.027320 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:03:38.038 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:03:38.038698 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:03:48.221 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:03:48.221152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:03:48.235 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:03:48.235769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:03:58.322 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:03:58.321742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:03:58.335 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:03:58.334839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:04:08.512 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:04:08.512131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:04:08.525 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:04:08.524731 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:04:18.591 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:04:18.590561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:04:18.608 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:04:18.608163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:04:28.677 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:04:28.676619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:04:28.688 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:04:28.688617 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:04:38.784 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:04:38.783720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:04:38.811 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:04:38.811048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:04:48.887 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:04:48.886417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:04:48.898 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:04:48.898186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:04:58.969 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:04:58.968839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:04:58.983 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:04:58.982585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:05:09.045 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:05:09.044562 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:05:09.058 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:05:09.057860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:05:19.144 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:05:19.144174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:05:19.155 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:05:19.154623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:05:29.339 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:05:29.338244 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:05:29.351 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:05:29.350575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:05:39.419 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:05:39.419303 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:05:39.432 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:05:39.432278 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:05:49.504 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:05:49.503199 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:05:49.522 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:05:49.517954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:05:59.590 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:05:59.589974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:05:59.604 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:05:59.603473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:06:09.699 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:06:09.699042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:06:09.711 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:06:09.711534 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:06:19.817 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:06:19.816862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:06:19.832 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:06:19.831615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:06:29.998 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:06:29.997072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:06:30.008 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:06:30.008333 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:06:40.062 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:06:40.061356 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:06:40.223 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:06:40.072736 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:06:50.139 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:06:50.138755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:06:50.149 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:06:50.148984 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:07:00.219 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:07:00.219048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:07:00.234 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:07:00.230565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:07:10.342 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:07:10.341876 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:07:10.356 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:07:10.356060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:07:20.433 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:07:20.431134 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:07:20.444 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:07:20.443709 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:07:30.510 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:07:30.510047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:07:30.523 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:07:30.522133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:07:40.590 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:07:40.589673 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:07:40.603 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:07:40.603394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:07:50.697 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:07:50.695568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:07:50.708 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:07:50.707881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:08:00.800 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:08:00.799632 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:08:00.812 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:08:00.811814 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:08:10.931 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:08:10.930384 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:08:10.944 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:08:10.944267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:08:21.061 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:08:21.060691 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:08:21.072 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:08:21.072496 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:08:31.199 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:08:31.198102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:08:31.214 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:08:31.213478 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:08:41.340 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:08:41.339229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:08:41.351 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:08:41.351078 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:08:51.417 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:08:51.416115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:08:51.429 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:08:51.428884 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:09:01.116 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:09:01.116015 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-08T00:09:01.528 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:09:01.527583 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:09:01.539 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:09:01.538833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:09:01.779 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:09:01.778549 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T00:09:01.770+0000 7f022e4ec380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T00:09:01.780 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T00:09:01.770+0000 7f022e4ec380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T00:09:01.780 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T00:09:01.770+0000 7f022e4ec380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T00:09:01.780 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-08T00:09:11.606 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:09:11.604778 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:09:11.613 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:09:11.612916 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:09:21.690 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:09:21.689493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:09:21.703 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:09:21.702666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:09:31.770 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:09:31.769487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:09:31.782 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:09:31.782310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:09:41.854 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:09:41.853720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:09:41.913 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:09:41.912511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:09:51.989 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:09:51.989421 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:09:52.003 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:09:52.000891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:10:02.103 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:10:02.096489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:10:02.108 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:10:02.106243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:10:12.173 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:10:12.172873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:10:12.187 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:10:12.187330 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:10:22.311 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:10:22.309535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:10:22.323 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:10:22.322940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:10:32.434 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:10:32.427128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:10:32.446 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:10:32.442334 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:10:42.547 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:10:42.547211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:10:42.559 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:10:42.559192 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:10:52.624 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:10:52.623448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:10:52.637 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:10:52.637259 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:11:02.827 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:11:02.826640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:11:02.836 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:11:02.835758 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:11:12.951 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:11:12.950321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:11:12.963 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:11:12.963126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:11:23.060 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:11:23.059917 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:11:23.075 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:11:23.074843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:11:33.188 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:11:33.187655 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:11:33.200 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:11:33.199726 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:11:43.525 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:11:43.525207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:11:43.540 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:11:43.539979 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:11:53.598 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:11:53.596687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:11:53.610 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:11:53.606343 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:12:03.702 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:12:03.702043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:12:03.713 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:12:03.712708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:12:13.788 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:12:13.788266 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:12:13.799 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:12:13.799044 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:12:23.872 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:12:23.871783 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:12:23.885 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:12:23.885212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:12:33.958 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:12:33.957231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:12:33.969 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:12:33.969079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:12:44.070 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:12:44.069412 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:12:44.089 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:12:44.086110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:12:54.159 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:12:54.155942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:12:54.229 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:12:54.228735 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:13:04.337 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:13:04.336977 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:13:04.349 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:13:04.349621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:13:14.450 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:13:14.448797 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:13:14.461 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:13:14.460857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:13:24.525 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:13:24.524777 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:13:24.537 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:13:24.536909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:13:34.624 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:13:34.623764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:13:34.679 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:13:34.679174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:13:44.751 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:13:44.750945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:13:44.810 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:13:44.809415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:13:54.903 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:13:54.899165 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:13:54.968 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:13:54.967949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:14:05.072 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:14:05.071172 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:14:05.084 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:14:05.084010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:14:15.166 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:14:15.165554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:14:15.201 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:14:15.201233 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:14:25.268 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:14:25.266887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:14:25.320 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:14:25.319670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:14:35.450 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:14:35.449319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:14:35.462 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:14:35.461404 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:14:45.553 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:14:45.552617 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:14:45.565 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:14:45.564911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:14:55.692 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:14:55.691886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:14:55.705 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:14:55.704917 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:15:05.763 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:15:05.761276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:15:05.775 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:15:05.774700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:15:15.840 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:15:15.839796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:15:15.855 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:15:15.853972 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:15:25.956 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:15:25.955739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:15:25.969 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:15:25.969497 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:15:36.036 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:15:36.036058 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:15:36.049 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:15:36.048769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:15:46.150 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:15:46.149837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:15:46.161 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:15:46.161549 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:15:56.236 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:15:56.231559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:15:56.291 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:15:56.291394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:16:06.366 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:16:06.365997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:16:06.424 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:16:06.423781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:16:16.511 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:16:16.511052 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:16:16.526 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:16:16.526097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:16:26.630 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:16:26.629560 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:16:26.641 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:16:26.640612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:16:36.731 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:16:36.730805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:16:36.745 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:16:36.745357 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:16:46.818 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:16:46.818017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:16:46.831 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:16:46.831234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:16:56.936 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:16:56.936032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:16:56.950 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:16:56.949905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:17:07.081 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:17:07.081472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:17:07.093 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:17:07.093588 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:17:17.303 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:17:17.302986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:17:17.315 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:17:17.315366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:17:27.414 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:17:27.413888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:17:27.427 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:17:27.427841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:17:37.490 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:17:37.490058 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:17:37.516 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:17:37.515447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:17:47.621 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:17:47.620345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:17:48.195 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:17:47.632392 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:17:57.731 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:17:57.730930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:17:57.743 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:17:57.743530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:18:07.865 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:18:07.865378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:18:07.876 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:18:07.876249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:18:17.939 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:18:17.939254 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:18:17.949 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:18:17.949362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:18:28.026 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:18:28.026387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:18:28.038 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:18:28.038021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:18:38.104 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:18:38.104265 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:18:38.152 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:18:38.151863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:18:48.264 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:18:48.263717 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:18:48.278 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:18:48.278561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:18:58.349 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:18:58.349021 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:18:58.415 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:18:58.415129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:19:08.484 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:19:08.484210 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:19:08.496 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:19:08.496443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:19:18.595 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:19:18.594772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:19:18.607 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:19:18.606999 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:19:28.771 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:19:28.771316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:19:28.782 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:19:28.781871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:19:38.853 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:19:38.853104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:19:38.866 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:19:38.866579 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:19:48.940 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:19:48.940319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:19:48.954 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:19:48.954198 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:19:57.145 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:19:57.144983 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-08T00:19:57.793 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:19:57.793011 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T00:19:57.782+0000 7f1deb9f1380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T00:19:57.793 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T00:19:57.782+0000 7f1deb9f1380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T00:19:57.794 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T00:19:57.782+0000 7f1deb9f1380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T00:19:57.794 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-08T00:19:59.017 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:19:59.017121 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:19:59.029 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:19:59.029374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:20:09.127 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:20:09.126834 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:20:09.137 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:20:09.137150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:20:19.242 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:20:19.242343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:20:19.254 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:20:19.254974 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:20:29.323 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:20:29.323619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:20:29.334 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:20:29.334954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:20:39.425 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:20:39.425059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:20:39.439 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:20:39.438973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:20:49.522 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:20:49.521998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:20:49.534 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:20:49.534136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:20:59.604 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:20:59.604488 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:20:59.658 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:20:59.658002 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:21:09.741 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:21:09.740818 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:21:09.793 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:21:09.793033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:21:19.859 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:21:19.858427 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:21:19.909 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:21:19.909221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:21:29.985 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:21:29.984461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:21:30.043 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:21:30.043117 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:21:40.104 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:21:40.104249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:21:40.120 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:21:40.120171 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:21:50.187 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:21:50.183646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:21:50.194 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:21:50.194780 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:22:00.263 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:22:00.263425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:22:00.273 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:22:00.273060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:22:10.351 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:22:10.351370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:22:10.365 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:22:10.365018 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:22:20.451 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:22:20.451079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:22:20.467 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:22:20.467013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:22:30.540 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:22:30.539907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:22:30.635 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:22:30.634587 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:22:40.736 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:22:40.735907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:22:40.750 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:22:40.747398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:22:50.843 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:22:50.842637 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:22:50.856 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:22:50.856321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:23:00.964 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:23:00.963943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:23:00.977 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:23:00.977510 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:23:11.082 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:23:11.081572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:23:11.093 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:23:11.093370 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:23:21.206 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:23:21.206063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:23:21.220 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:23:21.220029 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:23:31.362 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:23:31.361874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:23:31.375 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:23:31.375054 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:23:41.449 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:23:41.449143 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:23:41.462 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:23:41.462123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:23:51.568 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:23:51.568397 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:23:51.582 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:23:51.581993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:24:01.686 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:24:01.686009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:24:01.696 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:24:01.696774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:24:11.789 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:24:11.788572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:24:11.800 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:24:11.800446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:24:21.883 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:24:21.883232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:24:21.957 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:24:21.957059 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:24:32.013 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:24:32.013453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:24:32.025 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:24:32.025321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:24:42.104 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:24:42.103951 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:24:42.114 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:24:42.114585 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:24:52.243 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:24:52.242574 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:24:52.257 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:24:52.257204 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:25:02.368 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:25:02.366732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:25:02.380 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:25:02.380511 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:25:12.449 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:25:12.448686 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:25:12.499 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:25:12.499272 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:25:22.602 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:25:22.601534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:25:22.614 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:25:22.614494 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:25:32.691 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:25:32.690629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:25:32.704 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:25:32.704409 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:25:42.827 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:25:42.826620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:25:42.837 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:25:42.837292 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:25:52.903 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:25:52.903557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:25:52.918 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:25:52.918276 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:26:03.022 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:26:03.021829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:26:03.034 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:26:03.035019 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:26:13.099 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:26:13.099163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:26:13.111 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:26:13.110816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:26:23.219 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:26:23.219072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:26:23.232 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:26:23.232042 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:26:33.309 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:26:33.308470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:26:33.320 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:26:33.320464 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:26:43.410 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:26:43.409366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:26:43.422 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:26:43.421967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:26:53.498 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:26:53.497610 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:26:53.510 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:26:53.509378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:27:03.578 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:27:03.577543 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:27:03.589 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:27:03.589250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:27:13.690 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:27:13.690233 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:27:13.702 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:27:13.702667 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:27:23.778 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:27:23.778261 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:27:23.830 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:27:23.830342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:27:33.927 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:27:33.927179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:27:33.939 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:27:33.939568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:27:44.006 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:27:44.006238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:27:44.017 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:27:44.016993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:27:54.121 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:27:54.121571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:27:54.136 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:27:54.136090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:28:04.227 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:28:04.227505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:28:04.240 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:28:04.240507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:28:14.317 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:28:14.317176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:28:14.329 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:28:14.328850 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:28:24.405 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:28:24.404483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:28:24.416 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:28:24.416332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:28:34.530 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:28:34.529815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:28:34.545 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:28:34.544385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:28:44.668 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:28:44.667036 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:28:44.681 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:28:44.681265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:28:54.795 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:28:54.794776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:28:54.807 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:28:54.807184 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:29:04.884 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:29:04.884012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:29:04.896 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:29:04.896071 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:29:15.009 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:29:15.008499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:29:15.021 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:29:15.021425 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:29:25.108 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:29:25.108095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:29:25.166 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:29:25.165444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:29:35.251 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:29:35.250522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:29:35.259 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:29:35.258925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:29:45.383 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:29:45.382505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:29:45.397 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:29:45.397621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:29:55.459 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:29:55.458411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:29:55.551 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:29:55.550989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:30:05.612 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:30:05.611086 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:30:05.624 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:30:05.624177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:30:15.721 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:30:15.721218 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:30:15.732 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:30:15.732590 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:30:25.831 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:30:25.831083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:30:25.844 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:30:25.844787 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:30:35.937 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:30:35.937497 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:30:35.950 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:30:35.949902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:30:46.022 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:30:46.021841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:30:46.035 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:30:46.034610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:30:56.155 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:30:56.155006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:30:56.168 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:30:56.168121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:31:06.235 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:31:06.234750 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:31:06.247 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:31:06.247323 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:31:16.331 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:31:16.331289 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:31:16.341 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:31:16.341551 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:31:26.407 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:31:26.406733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:31:26.432 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:31:26.431759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:31:36.541 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:31:36.540732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:31:36.552 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:31:36.552308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:31:46.621 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:31:46.620979 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:31:46.632 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:31:46.632688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:31:56.729 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:31:56.727396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:31:56.747 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:31:56.742830 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:32:06.836 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:32:06.836338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:32:06.848 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:32:06.848436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:32:16.949 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:32:16.948473 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:32:16.963 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:32:16.962692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:32:27.035 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:32:27.034489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:32:27.048 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:32:27.047954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:32:37.116 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:32:37.115667 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:32:37.130 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:32:37.130397 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:32:47.227 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:32:47.226802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:32:47.240 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:32:47.240607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:32:57.320 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:32:57.313967 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:32:57.329 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:32:57.329301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:33:07.391 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:33:07.391310 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:33:07.405 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:33:07.405611 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:33:17.480 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:33:17.479930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:33:17.493 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:33:17.493737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:33:27.598 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:33:27.598270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:33:27.610 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:33:27.610268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:33:37.707 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:33:37.706854 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:33:37.718 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:33:37.718717 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:33:47.821 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:33:47.821161 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:33:47.835 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:33:47.834962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:33:57.932 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:33:57.932501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:33:57.944 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:33:57.944340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:34:08.014 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:34:08.014107 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:34:08.027 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:34:08.027295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:34:18.106 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:34:18.105766 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:34:18.117 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:34:18.117562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:34:28.255 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:34:28.255055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:34:28.268 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:34:28.268168 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:34:38.333 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:34:38.332645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:34:38.344 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:34:38.344246 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:34:48.466 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:34:48.465656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:34:48.479 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:34:48.479643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:34:58.577 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:34:58.576650 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:34:58.588 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:34:58.588650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:35:08.700 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:35:08.699789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:35:08.713 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:35:08.713473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:35:18.822 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:35:18.821751 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:35:18.833 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:35:18.832807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:35:28.934 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:35:28.933836 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:35:28.949 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:35:28.949047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:35:39.067 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:35:39.066418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:35:39.079 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:35:39.079161 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:35:49.206 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:35:49.205556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:35:49.214 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:35:49.214573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:35:59.326 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:35:59.325942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:35:59.338 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:35:59.337672 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:36:09.432 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:36:09.431819 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:36:09.443 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:36:09.443485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:36:19.514 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:36:19.514350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:36:19.526 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:36:19.526412 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:36:29.613 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:36:29.613280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:36:29.682 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:36:29.681880 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:36:37.802 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:36:37.801214 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-08T00:36:38.492 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:36:38.490866 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T00:36:38.482+0000 7fd5814c5380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T00:36:38.492 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T00:36:38.482+0000 7fd5814c5380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T00:36:38.492 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T00:36:38.482+0000 7fd5814c5380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T00:36:38.493 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-08T00:36:39.980 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:36:39.979747 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:36:39.994 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:36:39.993621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:36:50.102 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:36:50.102287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:36:50.113 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:36:50.113219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:37:00.172 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:37:00.171360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:37:00.211 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:37:00.211426 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:37:10.334 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:37:10.333991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:37:10.347 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:37:10.346877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:37:20.429 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:37:20.428903 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:37:20.481 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:37:20.480857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:37:30.605 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:37:30.605445 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:37:30.616 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:37:30.616295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:37:40.694 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:37:40.693711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:37:40.707 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:37:40.706715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:37:50.773 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:37:50.773133 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:37:50.786 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:37:50.785660 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:38:00.862 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:38:00.861599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:38:00.873 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:38:00.873533 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:38:10.944 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:38:10.942263 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:38:10.953 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:38:10.953389 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:38:21.015 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:38:21.014602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:38:21.028 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:38:21.027839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:38:31.138 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:38:31.137375 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:38:31.150 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:38:31.149996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:38:41.212 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:38:41.211350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:38:41.230 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:38:41.229940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:38:51.329 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:38:51.329077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:38:51.616 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:38:51.341285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:39:01.457 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:39:01.456079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:39:01.469 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:39:01.469175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:39:11.582 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:39:11.582168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:39:11.598 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:39:11.597978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:39:21.707 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:39:21.707031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:39:21.719 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:39:21.718463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:39:31.817 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:39:31.816755 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:39:31.831 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:39:31.830751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:39:41.945 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:39:41.944985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:39:41.959 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:39:41.959383 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:39:52.031 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:39:52.031008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:39:52.043 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:39:52.042927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:40:02.111 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:40:02.110666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:40:02.122 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:40:02.122312 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:40:12.186 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:40:12.185904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:40:12.197 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:40:12.197552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:40:22.273 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:40:22.268601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:40:22.281 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:40:22.281312 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:40:32.353 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:40:32.353130 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:40:32.403 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:40:32.403506 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:40:42.497 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:40:42.497286 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:40:42.510 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:40:42.510429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:40:52.588 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:40:52.587887 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:40:52.640 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:40:52.639714 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:41:02.712 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:41:02.712000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:41:02.724 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:41:02.724148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:41:12.791 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:41:12.790511 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:41:12.806 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:41:12.806003 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:41:22.937 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:41:22.936387 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:41:22.950 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:41:22.949812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:41:33.037 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:41:33.036316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:41:33.049 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:41:33.048558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:41:43.130 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:41:43.129638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:41:43.141 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:41:43.141291 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:41:53.210 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:41:53.209713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:41:53.258 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:41:53.258174 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:42:03.326 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:42:03.325807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:42:03.339 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:42:03.338923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:42:13.406 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:42:13.405752 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:42:13.454 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:42:13.454347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:42:23.525 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:42:23.524078 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:42:23.540 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:42:23.539563 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:42:33.641 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:42:33.640443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:42:33.655 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:42:33.654326 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:42:43.732 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:42:43.730405 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:42:43.829 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:42:43.828603 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:42:53.930 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:42:53.929638 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:42:53.944 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:42:53.944284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:43:04.018 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:43:04.017470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:43:04.029 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:43:04.029422 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:43:14.150 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:43:14.147293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:43:14.159 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:43:14.159010 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:43:24.227 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:43:24.226976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:43:24.243 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:43:24.242363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:43:34.356 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:43:34.355504 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:43:34.371 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:43:34.371491 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:43:44.472 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:43:44.471358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:43:44.486 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:43:44.486131 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:43:54.624 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:43:54.623318 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:43:54.637 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:43:54.636823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:44:04.738 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:44:04.738191 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:44:04.751 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:44:04.751104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:44:14.848 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:44:14.847796 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:44:14.857 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:44:14.856594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:44:24.996 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:44:24.995611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:44:25.008 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:44:25.007663 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:44:35.112 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:44:35.111773 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:44:35.123 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:44:35.123273 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:44:45.210 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:44:45.209790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:44:45.223 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:44:45.222676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:44:55.315 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:44:55.314344 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:44:55.333 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:44:55.333186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:45:05.420 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:45:05.419967 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:45:05.471 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:45:05.471082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:45:15.563 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:45:15.563008 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:45:15.576 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:45:15.575642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:45:25.650 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:45:25.649656 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:45:25.713 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:45:25.712779 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:45:35.822 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:45:35.821549 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:45:35.835 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:45:35.834508 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:45:45.924 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:45:45.924056 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:45:45.939 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:45:45.938675 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:45:56.003 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:45:56.002782 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:45:56.016 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:45:56.015650 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:46:06.081 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:46:06.080551 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:46:06.092 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:46:06.092237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:46:16.180 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:46:16.179817 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:46:16.216 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:46:16.216138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:46:26.385 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:46:26.384069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:46:26.394 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:46:26.392715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:46:36.458 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:46:36.458273 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:46:36.525 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:46:36.525160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:46:46.664 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:46:46.664018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:46:46.677 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:46:46.677100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:46:56.753 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:46:56.753102 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:46:56.885 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:46:56.765067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:47:06.832 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:47:06.830788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:47:06.882 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:47:06.881856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:47:16.981 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:47:16.980816 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:47:16.994 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:47:16.993912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:47:27.102 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:47:27.101618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:47:27.113 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:47:27.113535 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:47:37.237 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:47:37.236646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:47:37.246 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:47:37.246017 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:47:47.318 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:47:47.317831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:47:47.331 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:47:47.330463 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:47:57.441 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:47:57.441206 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:47:57.456 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:47:57.456021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:48:07.550 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:48:07.549572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:48:07.563 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:48:07.562952 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:48:17.652 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:48:17.651329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:48:17.723 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:48:17.723083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:48:27.851 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:48:27.850969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:48:27.864 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:48:27.863958 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:48:37.950 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:48:37.950217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:48:37.963 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:48:37.963125 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:48:48.023 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:48:48.022159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:48:48.065 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:48:48.064684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:48:58.171 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:48:58.170354 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:48:58.181 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:48:58.180761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:49:08.248 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:49:08.247792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:49:08.261 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:49:08.260821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:49:18.336 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:49:18.336235 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:49:18.351 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:49:18.350798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:49:28.412 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:49:28.411239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:49:28.425 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:49:28.425503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:49:38.499 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:49:38.499249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:49:38.512 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:49:38.511886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:49:48.578 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:49:48.578087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:49:48.591 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:49:48.591309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:49:58.663 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:49:58.663109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:49:58.677 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:49:58.676873 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:50:08.748 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:50:08.747935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:50:08.760 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:50:08.760315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:50:18.872 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:50:18.871768 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:50:18.986 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:50:18.884851 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:50:28.969 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:50:28.968460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:50:28.982 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:50:28.982415 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:50:39.085 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:50:39.084906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:50:39.098 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:50:39.097377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:50:49.217 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:50:49.216969 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:50:49.232 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:50:49.231941 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:50:59.316 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:50:59.315825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:50:59.346 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:50:59.345751 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:51:09.460 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:51:09.460185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:51:09.472 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:51:09.472359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:51:19.571 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:51:19.571069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:51:19.582 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:51:19.582779 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:51:29.695 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:51:29.695363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:51:29.707 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:51:29.707556 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:51:39.773 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:51:39.773337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:51:39.787 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:51:39.787024 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:51:49.885 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:51:49.885238 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:51:49.897 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:51:49.897378 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:51:59.967 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:51:59.966795 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:51:59.980 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:51:59.980123 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:52:10.052 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:52:10.052173 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:52:10.063 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:52:10.062877 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:52:20.168 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:52:20.167772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:52:20.180 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:52:20.180723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:52:30.330 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:52:30.329874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:52:30.342 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:52:30.342195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:52:40.410 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:52:40.410000 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:52:40.423 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:52:40.422961 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:52:50.522 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:52:50.522255 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:52:50.533 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:52:50.533077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:53:00.616 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:53:00.615997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:53:00.683 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:53:00.682800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:53:10.754 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:53:10.754198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:53:10.767 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:53:10.767343 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:53:18.504 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:53:18.503859 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-08T00:53:19.141 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:53:19.140699 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T00:53:19.130+0000 7faf47f8f380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T00:53:19.141 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T00:53:19.130+0000 7faf47f8f380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T00:53:19.142 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T00:53:19.130+0000 7faf47f8f380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T00:53:19.142 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-08T00:53:20.869 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:53:20.868820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:53:20.881 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:53:20.881315 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:53:30.989 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:53:30.988880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:53:31.001 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:53:31.001352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:53:41.113 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:53:41.113396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:53:41.126 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:53:41.126361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:53:51.232 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:53:51.232063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:53:51.245 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:53:51.244833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:54:01.362 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:54:01.361129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:54:01.373 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:54:01.373816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:54:11.475 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:54:11.475193 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:54:11.490 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:54:11.490252 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:54:21.595 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:54:21.594360 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:54:21.876 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:54:21.606153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:54:31.679 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:54:31.678886 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:54:31.692 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:54:31.692526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:54:41.800 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:54:41.799599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:54:41.811 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:54:41.811207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:54:51.894 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:54:51.893946 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:54:51.909 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:54:51.905403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:55:02.045 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:55:02.045151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:55:02.055 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:55:02.054938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:55:12.146 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:55:12.145725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:55:12.157 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:55:12.156860 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:55:22.240 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:55:22.240135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:55:22.251 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:55:22.251458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:55:32.487 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:55:32.486825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:55:32.501 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:55:32.501004 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:55:42.604 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:55:42.603844 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:55:42.613 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:55:42.613279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:55:52.682 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:55:52.681343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:55:52.697 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:55:52.697076 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:56:02.957 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:56:02.957124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:56:02.970 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:56:02.968490 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:56:13.054 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:56:13.053545 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:56:13.066 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:56:13.066607 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:56:23.129 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:56:23.128449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:56:23.190 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:56:23.189713 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:56:33.280 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:56:33.280001 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:56:33.295 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:56:33.294913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:56:43.368 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:56:43.367487 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:56:43.380 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:56:43.380107 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:56:53.491 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:56:53.491148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:56:53.505 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:56:53.505397 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:57:03.611 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:57:03.610799 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:57:03.622 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:57:03.622503 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:57:13.714 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:57:13.714560 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:57:13.731 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:57:13.731132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:57:23.848 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:57:23.848468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:57:23.857 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:57:23.857639 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:57:33.920 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:57:33.920602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:57:33.973 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:57:33.973173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:57:44.046 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:57:44.046701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:57:44.059 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:57:44.060013 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:57:54.137 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:57:54.133120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:57:54.149 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:57:54.148821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:58:04.252 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:58:04.251744 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:58:04.262 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:58:04.262761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:58:14.333 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:58:14.332745 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:58:14.399 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:58:14.399488 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:58:24.504 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:58:24.504518 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:58:24.516 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:58:24.516875 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:58:34.593 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:58:34.593279 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:58:34.657 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:58:34.656853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:58:44.764 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:58:44.763904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:58:44.776 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:58:44.776235 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:58:54.897 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:58:54.897025 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:58:54.910 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:58:54.910456 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:59:04.968 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:59:04.968358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:59:05.095 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:59:04.980879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:59:15.080 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:59:15.079530 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:59:15.092 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:59:15.092624 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:59:25.189 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:59:25.189559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:59:25.202 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:59:25.202060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:59:35.302 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:59:35.302164 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:59:35.315 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:59:35.315427 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:59:45.406 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:59:45.406051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:59:45.418 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:59:45.418927 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:59:55.483 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:59:55.483226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T00:59:55.496 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 00:59:55.496318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:00:05.564 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:00:05.563873 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:00:05.577 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:00:05.577112 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:00:15.700 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:00:15.700069 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:00:15.713 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:00:15.713239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:00:25.820 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:00:25.820156 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:00:25.854 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:00:25.854425 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:00:35.964 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:00:35.964432 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:00:35.977 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:00:35.977436 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:00:46.040 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:00:46.039899 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:00:46.051 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:00:46.050979 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:00:56.120 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:00:56.119839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:00:56.131 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:00:56.131298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:01:06.247 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:01:06.247304 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:01:06.262 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:01:06.261853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:01:16.329 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:01:16.329162 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:01:16.342 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:01:16.342167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:01:26.480 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:01:26.479021 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:01:26.491 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:01:26.491111 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:01:36.589 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:01:36.589126 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:01:36.603 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:01:36.602684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:01:46.689 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:01:46.689240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:01:46.702 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:01:46.702173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:01:56.806 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:01:56.806009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:01:56.819 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:01:56.819281 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:02:06.929 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:02:06.929461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:02:06.942 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:02:06.942523 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:02:17.058 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:02:17.058333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:02:17.074 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:02:17.074543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:02:27.187 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:02:27.186884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:02:27.205 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:02:27.204379 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:02:37.282 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:02:37.282189 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:02:37.295 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:02:37.295127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:02:47.355 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:02:47.354185 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:02:47.406 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:02:47.405824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:02:57.493 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:02:57.492030 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:02:57.506 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:02:57.505896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:03:07.575 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:03:07.574938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:03:07.591 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:03:07.591682 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:03:17.692 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:03:17.691935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:03:17.709 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:03:17.709277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:03:27.811 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:03:27.811100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:03:27.823 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:03:27.823163 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:03:37.895 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:03:37.894293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:03:37.906 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:03:37.906641 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:03:48.006 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:03:48.006365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:03:48.020 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:03:48.020401 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:03:58.087 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:03:58.087049 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:03:58.100 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:03:58.100236 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:04:08.200 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:04:08.200290 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:04:08.213 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:04:08.213843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:04:18.286 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:04:18.286017 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:04:18.298 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:04:18.299051 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:04:28.430 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:04:28.427385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:04:28.439 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:04:28.439170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:04:38.520 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:04:38.520250 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:04:38.562 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:04:38.562553 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:04:48.657 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:04:48.656890 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:04:48.670 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:04:48.670773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:04:58.789 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:04:58.788981 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:04:58.928 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:04:58.802770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:05:08.913 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:05:08.913724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:05:09.495 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:05:08.927730 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:05:19.003 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:05:19.003005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:05:19.017 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:05:19.017995 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:05:29.211 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:05:29.211695 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:05:29.221 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:05:29.221689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:05:39.327 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:05:39.325437 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:05:39.335 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:05:39.335949 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:05:49.413 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:05:49.413563 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:05:49.473 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:05:49.473178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:05:59.555 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:05:59.555809 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:05:59.628 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:05:59.628008 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:06:09.711 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:06:09.711505 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:06:09.761 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:06:09.760776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:06:19.851 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:06:19.851029 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:06:19.861 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:06:19.861644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:06:29.932 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:06:29.932600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:06:29.945 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:06:29.945300 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:06:40.047 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:06:40.047533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:06:40.059 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:06:40.059652 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:06:50.160 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:06:50.159731 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:06:50.172 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:06:50.172186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:07:00.274 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:07:00.273827 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:07:00.286 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:07:00.286308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:07:10.384 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:07:10.384136 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:07:10.397 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:07:10.397759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:07:20.462 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:07:20.462776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:07:20.476 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:07:20.476439 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:07:30.550 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:07:30.550209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:07:30.562 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:07:30.561770 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:07:40.627 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:07:40.627703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:07:40.676 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:07:40.675766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:07:50.772 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:07:50.772083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:07:50.788 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:07:50.787914 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:08:00.856 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:08:00.855907 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:08:00.869 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:08:00.869419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:08:10.967 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:08:10.967325 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:08:10.981 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:08:10.982090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:08:21.090 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:08:21.089804 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:08:21.103 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:08:21.103045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:08:31.211 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:08:31.211043 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:08:31.224 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:08:31.224191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:08:41.327 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:08:41.327123 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:08:41.340 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:08:41.340790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:08:51.410 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:08:51.410828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:08:51.420 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:08:51.420595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:09:01.487 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:09:01.487613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:09:01.529 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:09:01.529818 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:09:11.645 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:09:11.645423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:09:11.860 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:09:11.658112 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:09:21.811 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:09:21.811046 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:09:21.821 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:09:21.821925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:09:31.888 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:09:31.888627 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:09:31.899 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:09:31.900067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:09:41.999 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:09:41.999699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:09:42.012 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:09:42.012659 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:09:52.114 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:09:52.112227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:09:52.121 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:09:52.121750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:09:59.149 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:09:59.149656 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-08T01:09:59.808 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:09:59.808323 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T01:09:59.798+0000 7fa548849380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T01:09:59.809 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T01:09:59.798+0000 7fa548849380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T01:09:59.809 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T01:09:59.798+0000 7fa548849380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T01:09:59.809 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-08T01:10:02.186 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:10:02.186857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:10:02.235 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:10:02.235971 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:10:12.306 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:10:12.306040 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:10:12.320 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:10:12.320671 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:10:22.463 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:10:22.463013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:10:22.473 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:10:22.474201 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:10:32.590 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:10:32.589950 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:10:32.601 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:10:32.601542 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:10:42.674 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:10:42.674382 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:10:42.686 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:10:42.686011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:10:52.763 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:10:52.763709 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:10:52.775 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:10:52.775903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:11:02.888 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:11:02.887385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:11:02.896 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:11:02.896773 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:11:12.987 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:11:12.987044 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:11:12.996 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:11:12.996492 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:11:23.066 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:11:23.066815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:11:23.080 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:11:23.080810 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:11:33.168 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:11:33.168533 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:11:33.180 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:11:33.180187 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:11:43.256 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:11:43.256168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:11:43.267 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:11:43.268267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:11:53.365 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:11:53.365599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:11:53.379 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:11:53.379693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:12:03.487 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:12:03.488079 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:12:03.501 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:12:03.501532 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:12:13.639 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:12:13.639467 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:12:13.650 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:12:13.650739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:12:23.756 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:12:23.756420 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:12:23.834 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:12:23.834349 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:12:33.918 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:12:33.917441 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:12:33.977 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:12:33.976764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:12:44.096 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:12:44.096973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:12:44.175 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:12:44.111597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:12:54.243 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:12:54.243210 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:12:54.256 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:12:54.255962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:13:04.355 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:13:04.355681 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:13:04.364 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:13:04.365352 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:13:14.473 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:13:14.473410 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:13:14.487 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:13:14.487454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:13:24.633 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:13:24.633952 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:13:24.646 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:13:24.646944 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:13:34.754 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:13:34.754601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:13:34.765 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:13:34.766047 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:13:44.841 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:13:44.841406 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:13:44.908 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:13:44.908170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:13:55.007 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:13:55.007599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:13:55.018 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:13:55.019320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:14:05.096 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:14:05.096535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:14:05.172 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:14:05.172490 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:14:15.287 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:14:15.287132 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:14:15.301 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:14:15.301795 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:14:25.385 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:14:25.385603 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:14:25.397 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:14:25.397889 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:14:35.463 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:14:35.463125 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:14:35.475 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:14:35.475963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:14:45.589 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:14:45.589691 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:14:45.604 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:14:45.604369 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:14:55.672 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:14:55.673211 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:14:55.685 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:14:55.686081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:15:05.836 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:15:05.836250 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:15:05.849 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:15:05.849536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:15:15.948 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:15:15.948881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:15:15.970 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:15:15.971178 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:15:26.080 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:15:26.080708 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:15:26.137 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:15:26.137886 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:15:36.434 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:15:36.434770 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:15:36.461 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:15:36.461896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:15:46.564 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:15:46.564333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:15:46.576 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:15:46.577133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:15:56.703 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:15:56.703762 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:15:56.716 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:15:56.717053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:16:06.877 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:16:06.877154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:16:06.888 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:16:06.888939 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:16:17.153 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:16:17.152946 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:16:17.165 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:16:17.165874 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:16:27.314 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:16:27.314708 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:16:27.327 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:16:27.327870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:16:37.436 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:16:37.436236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:16:37.451 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:16:37.451507 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:16:47.535 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:16:47.535851 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:16:47.549 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:16:47.549856 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:16:57.652 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:16:57.651895 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:16:57.663 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:16:57.664135 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:17:07.731 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:17:07.731523 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:17:07.746 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:17:07.747225 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:17:17.822 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:17:17.822701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:17:17.876 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:17:17.876332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:17:27.939 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:17:27.939212 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:17:27.951 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:17:27.951841 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:17:38.032 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:17:38.032874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:17:38.045 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:17:38.044799 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:17:48.127 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:17:48.127300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:17:48.139 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:17:48.139040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:17:58.245 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:17:58.245119 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:17:58.258 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:17:58.258527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:18:08.363 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:18:08.363343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:18:08.374 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:18:08.374837 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:18:18.480 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:18:18.480363 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:18:18.492 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:18:18.493167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:18:28.609 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:18:28.609584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:18:28.622 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:18:28.623389 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:18:38.686 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:18:38.686391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:18:38.695 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:18:38.696064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:18:48.818 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:18:48.818730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:18:48.829 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:18:48.829645 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:18:58.896 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:18:58.896598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:18:58.909 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:18:58.910221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:19:09.016 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:19:09.016426 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:19:09.026 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:19:09.026903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:19:19.149 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:19:19.149785 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:19:19.161 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:19:19.162517 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:19:29.278 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:19:29.278051 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:19:29.289 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:19:29.289632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:19:39.368 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:19:39.368442 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:19:39.378 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:19:39.378806 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:19:49.486 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:19:49.486923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:19:49.499 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:19:49.500116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:19:59.598 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:19:59.597032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:19:59.608 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:19:59.608469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:20:09.755 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:20:09.755550 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:20:09.766 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:20:09.767321 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:20:19.894 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:20:19.894597 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:20:19.905 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:20:19.906110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:20:29.991 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:20:29.991527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:20:30.004 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:20:30.004438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:20:40.074 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:20:40.074776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:20:40.085 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:20:40.086254 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:20:50.194 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:20:50.193855 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:20:50.211 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:20:50.208644 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:21:00.308 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:21:00.308798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:21:00.321 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:21:00.322121 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:21:10.420 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:21:10.420997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:21:10.434 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:21:10.434700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:21:20.509 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:21:20.509995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:21:20.520 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:21:20.521016 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:21:30.656 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:21:30.656820 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:21:30.671 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:21:30.668612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:21:40.806 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:21:40.806618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:21:40.815 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:21:40.816482 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:21:50.929 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:21:50.929448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:21:50.941 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:21:50.942148 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:22:01.069 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:22:01.069384 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:22:01.080 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:22:01.081083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:22:11.154 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:22:11.154512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:22:11.166 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:22:11.167264 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:22:21.232 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:22:21.232677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:22:21.246 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:22:21.246779 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:22:31.318 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:22:31.318629 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:22:31.333 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:22:31.333301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:22:41.409 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:22:41.410089 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:22:41.434 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:22:41.435050 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:22:51.503 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:22:51.503779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:22:51.516 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:22:51.517284 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:23:01.584 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:23:01.584615 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:23:01.650 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:23:01.650804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:23:11.769 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:23:11.769588 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:23:11.781 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:23:11.781621 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:23:21.852 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:23:21.852923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:23:21.864 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:23:21.864472 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:23:31.928 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:23:31.928377 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:23:31.938 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:23:31.939240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:23:42.035 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:23:42.035307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:23:42.047 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:23:42.047739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:23:52.207 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:23:52.208178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:23:52.218 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:23:52.219332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:24:02.306 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:24:02.306741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:24:02.320 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:24:02.320757 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:24:12.403 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:24:12.403749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:24:12.413 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:24:12.414443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:24:22.526 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:24:22.526908 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:24:22.541 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:24:22.541177 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:24:32.655 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:24:32.655879 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:24:32.667 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:24:32.668182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:24:42.731 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:24:42.731460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:24:42.744 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:24:42.744985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:24:52.822 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:24:52.822027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:24:52.835 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:24:52.835776 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:25:02.918 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:25:02.919005 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:25:02.967 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:25:02.967804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:25:13.030 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:25:13.030524 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:25:13.043 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:25:13.043605 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:25:23.151 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:25:23.151604 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:25:23.165 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:25:23.165538 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:25:33.233 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:25:33.234223 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:25:33.247 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:25:33.248080 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:25:43.343 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:25:43.343649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:25:43.355 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:25:43.356104 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:25:53.452 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:25:53.452696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:25:53.465 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:25:53.465781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:26:03.589 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:26:03.590221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:26:03.598 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:26:03.599355 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:26:13.662 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:26:13.662178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:26:13.674 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:26:13.675128 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:26:23.790 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:26:23.790802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:26:23.806 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:26:23.805806 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:26:33.871 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:26:33.871788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:26:33.881 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:26:33.882341 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:26:39.818 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:26:39.818880 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-08T01:26:40.489 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:26:40.488940 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T01:26:40.479+0000 7fca3ee23380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T01:26:40.490 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T01:26:40.479+0000 7fca3ee23380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T01:26:40.490 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T01:26:40.479+0000 7fca3ee23380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T01:26:40.490 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-08T01:26:43.981 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:26:43.981595 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:26:43.990 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:26:43.991190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:26:54.057 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:26:54.058349 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:26:54.070 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:26:54.071561 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:27:04.161 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:27:04.161787 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:27:04.174 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:27:04.174623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:27:14.284 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:27:14.284596 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:27:14.297 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:27:14.298592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:27:24.375 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:27:24.375282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:27:24.451 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:27:24.451502 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:27:34.548 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:27:34.549229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:27:34.563 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:27:34.564088 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:27:44.641 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:27:44.641600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:27:44.654 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:27:44.654920 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:27:54.717 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:27:54.717527 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:27:54.725 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:27:54.726183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:28:04.813 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:28:04.813929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:28:04.827 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:28:04.828395 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:28:14.947 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:28:14.947376 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:28:14.959 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:28:14.959533 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:28:25.087 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:28:25.087572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:28:25.100 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:28:25.100807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:28:35.200 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:28:35.201296 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:28:35.214 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:28:35.213251 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:28:45.325 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:28:45.325846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:28:45.336 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:28:45.337164 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:28:55.455 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:28:55.456187 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:28:55.469 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:28:55.469564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:29:05.592 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:29:05.592230 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:29:05.607 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:29:05.607947 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:29:15.673 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:29:15.673519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:29:15.687 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:29:15.688239 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:29:25.792 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:29:25.793140 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:29:25.803 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:29:25.804240 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:29:35.873 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:29:35.874201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:29:35.886 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:29:35.886993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:29:46.002 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:29:46.003063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:29:46.017 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:29:46.017522 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:29:56.121 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:29:56.122128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:29:56.132 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:29:56.131912 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:30:06.197 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:30:06.198366 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:30:06.209 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:30:06.210583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:30:16.301 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:30:16.301877 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:30:16.314 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:30:16.314750 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:30:26.376 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:30:26.377167 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:30:26.388 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:30:26.388845 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:30:36.510 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:30:36.510918 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:30:36.523 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:30:36.523989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:30:46.640 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:30:46.641288 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:30:46.653 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:30:46.654060 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:30:56.764 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:30:56.765172 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:30:56.776 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:30:56.777327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:31:06.907 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:31:06.907697 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:31:06.920 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:31:06.920958 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:31:16.989 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:31:16.989639 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:31:17.030 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:31:17.030610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:31:27.136 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:31:27.136863 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:31:27.147 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:31:27.148307 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:31:37.212 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:31:37.212337 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:31:37.224 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:31:37.225465 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:31:47.324 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:31:47.325010 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:31:47.334 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:31:47.335311 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:31:57.435 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:31:57.435758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:31:57.450 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:31:57.448229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:32:07.514 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:32:07.514513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:32:07.525 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:32:07.526354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:32:17.591 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:32:17.591989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:32:17.602 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:32:17.603591 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:32:27.675 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:32:27.673698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:32:27.693 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:32:27.692824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:32:37.760 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:32:37.761237 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:32:37.775 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:32:37.775319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:32:47.848 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:32:47.849423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:32:47.858 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:32:47.859433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:32:57.955 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:32:57.956348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:32:57.969 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:32:57.969907 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:33:08.072 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:33:08.072418 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:33:08.083 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:33:08.084324 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:33:18.173 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:33:18.172701 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:33:18.184 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:33:18.184967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:33:28.296 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:33:28.296990 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:33:28.309 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:33:28.310680 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:33:38.379 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:33:38.379791 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:33:38.393 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:33:38.393815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:33:48.503 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:33:48.503559 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:33:48.516 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:33:48.517640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:33:58.596 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:33:58.597226 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:33:58.611 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:33:58.609911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:34:08.710 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:34:08.711004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:34:08.797 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:34:08.724954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:34:18.795 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:34:18.795628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:34:18.878 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:34:18.879481 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:34:28.951 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:34:28.952270 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:34:29.016 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:34:29.017419 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:34:39.123 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:34:39.123786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:34:39.137 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:34:39.138282 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:34:49.242 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:34:49.243080 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:34:49.253 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:34:49.254567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:34:59.386 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:34:59.386825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:34:59.398 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:34:59.399454 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:35:09.493 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:35:09.493554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:35:09.503 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:35:09.504528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:35:19.566 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:35:19.567305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:35:19.579 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:35:19.579951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:35:29.697 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:35:29.698144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:35:29.709 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:35:29.709993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:35:39.875 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:35:39.876519 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:35:39.886 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:35:39.887232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:35:49.964 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:35:49.965300 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:35:49.975 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:35:49.976057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:36:00.039 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:36:00.040599 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:36:00.050 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:36:00.051715 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:36:10.154 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:36:10.154666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:36:10.165 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:36:10.166760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:36:20.297 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:36:20.297906 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:36:20.308 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:36:20.309399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:36:30.365 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:36:30.365971 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:36:30.379 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:36:30.380012 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:36:40.441 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:36:40.442391 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:36:40.455 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:36:40.455992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:36:50.556 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:36:50.556811 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:36:50.571 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:36:50.572804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:37:00.689 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:37:00.689351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:37:00.702 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:37:00.702899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:37:10.770 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:37:10.771015 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:37:10.838 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:37:10.839101 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:37:20.947 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:37:20.947847 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:37:20.960 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:37:20.961301 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:37:31.078 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:37:31.078866 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:37:31.093 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:37:31.094539 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:37:41.236 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:37:41.236517 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:37:41.246 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:37:41.247473 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:37:51.323 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:37:51.324249 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:37:51.334 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:37:51.335601 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:38:01.411 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:38:01.411904 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:38:01.424 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:38:01.424813 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:38:11.530 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:38:11.530935 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:38:11.542 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:38:11.543237 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:38:21.620 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:38:21.621175 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:38:21.632 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:38:21.632858 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:38:31.705 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:38:31.705891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:38:31.719 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:38:31.719901 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:38:41.796 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:38:41.796857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:38:41.815 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:38:41.815793 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:38:51.888 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:38:51.888095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:38:51.897 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:38:51.898634 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:39:01.971 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:39:01.972392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:39:02.013 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:39:02.014066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:39:12.175 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:39:12.176429 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:39:12.188 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:39:12.189015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:39:22.309 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:39:22.310146 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:39:22.323 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:39:22.323998 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:39:32.397 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:39:32.398110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:39:32.409 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:39:32.410344 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:39:42.518 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:39:42.518692 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:39:42.532 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:39:42.533077 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:39:52.634 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:39:52.635026 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:39:52.645 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:39:52.646310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:40:02.751 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:40:02.751648 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:40:02.763 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:40:02.764136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:40:12.823 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:40:12.823905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:40:12.836 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:40:12.837337 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:40:22.921 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:40:22.921417 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:40:22.971 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:40:22.972118 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:40:33.074 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:40:33.075150 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:40:33.087 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:40:33.088165 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:40:43.200 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:40:43.200556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:40:43.211 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:40:43.212614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:40:53.294 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:40:53.294823 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:40:53.306 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:40:53.307796 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:41:03.417 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:41:03.417917 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:41:03.429 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:41:03.430084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:41:13.577 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:41:13.578493 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:41:13.628 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:41:13.629269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:41:23.767 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:41:23.768109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:41:23.779 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:41:23.780186 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:41:33.857 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:41:33.858131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:41:33.872 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:41:33.873760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:41:43.980 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:41:43.981529 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:41:43.991 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:41:43.992817 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:41:54.105 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:41:54.106151 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:41:54.120 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:41:54.120559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:42:04.243 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:42:04.242930 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:42:04.256 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:42:04.256724 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:42:14.352 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:42:14.353227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:42:14.364 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:42:14.365155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:42:24.429 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:42:24.430312 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:42:24.440 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:42:24.441399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:42:34.563 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:42:34.564085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:42:34.574 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:42:34.575253 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:42:44.674 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:42:44.674499 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:42:44.685 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:42:44.686192 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:42:54.751 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:42:54.752343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:42:54.767 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:42:54.766489 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:43:04.878 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:43:04.878685 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:43:04.890 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:43:04.891573 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:43:14.989 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:43:14.989711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:43:15.002 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:43:15.003600 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:43:20.496 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:43:20.496389 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-08T01:43:21.141 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:43:21.140681 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T01:43:21.127+0000 7eff356d9380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T01:43:21.141 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T01:43:21.127+0000 7eff356d9380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T01:43:21.141 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T01:43:21.127+0000 7eff356d9380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T01:43:21.142 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-08T01:43:25.104 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:43:25.105282 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:43:25.118 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:43:25.118567 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:43:35.191 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:43:35.192091 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:43:35.204 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:43:35.205138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:43:45.266 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:43:45.266707 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:43:45.278 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:43:45.279286 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:43:55.347 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:43:55.347461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:43:55.393 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:43:55.394268 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:44:05.490 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:44:05.491135 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:44:05.503 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:44:05.504196 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:44:15.603 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:44:15.604153 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:44:15.615 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:44:15.616739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:44:25.681 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:44:25.681883 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:44:25.693 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:44:25.693804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:44:35.774 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:44:35.774779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:44:35.785 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:44:35.785332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:44:45.852 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:44:45.853096 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:44:45.870 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:44:45.871170 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:44:55.978 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:44:55.978968 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:44:55.991 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:44:55.991882 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:45:06.097 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:45:06.097813 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:45:06.111 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:45:06.112504 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:45:16.237 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:45:16.238333 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:45:16.251 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:45:16.252119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:45:26.332 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:45:26.333509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:45:26.346 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:45:26.347764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:45:36.458 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:45:36.458459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:45:36.471 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:45:36.472319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:45:46.535 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:45:46.535832 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:45:46.547 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:45:46.548234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:45:56.673 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:45:56.673767 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:45:56.687 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:45:56.688160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:46:06.754 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:46:06.755584 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:46:06.765 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:46:06.766958 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:46:16.952 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:46:16.953157 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:46:17.015 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:46:17.015905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:46:27.114 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:46:27.114433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:46:27.134 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:46:27.133103 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:46:37.209 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:46:37.210103 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:46:37.251 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:46:37.251818 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:46:47.362 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:46:47.363071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:46:47.380 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:46:47.381191 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:46:57.493 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:46:57.493717 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:46:57.506 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:46:57.507760 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:47:07.579 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:47:07.578810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:47:07.758 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:47:07.591887 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:47:17.687 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:47:17.688160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:47:17.699 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:47:17.700774 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:47:27.823 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:47:27.823663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:47:27.842 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:47:27.839242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:47:37.899 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:47:37.900050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:47:37.910 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:47:37.911676 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:47:47.981 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:47:47.982705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:47:47.995 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:47:47.996693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:47:58.094 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:47:58.094643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:47:58.108 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:47:58.109353 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:48:08.180 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:48:08.181031 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:48:08.192 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:48:08.192909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:48:18.308 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:48:18.309541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:48:18.325 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:48:18.325815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:48:28.431 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:48:28.432004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:48:28.445 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:48:28.445809 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:48:38.539 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:48:38.539341 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:48:38.552 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:48:38.552881 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:48:48.638 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:48:48.635515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:48:48.694 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:48:48.694335 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:48:58.822 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:48:58.823500 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:48:58.833 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:48:58.834466 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:49:08.956 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:49:08.956468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:49:08.968 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:49:08.969638 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:49:19.043 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:49:19.043509 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:49:19.055 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:49:19.056394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:49:29.136 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:49:29.137058 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:49:29.249 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:49:29.250480 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:49:39.344 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:49:39.344864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:49:39.356 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:49:39.357807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:49:49.464 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:49:49.464910 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:49:49.479 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:49:49.480124 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:49:59.587 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:49:59.587916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:49:59.752 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:49:59.753265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:50:09.867 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:50:09.867989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:50:09.878 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:50:09.879151 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:50:19.980 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:50:19.981691 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:50:19.993 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:50:19.994138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:50:30.083 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:50:30.083452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:50:30.099 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:50:30.096159 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:50:40.165 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:50:40.166453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:50:40.222 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:50:40.223675 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:50:50.331 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:50:50.332307 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:50:50.343 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:50:50.343749 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:51:00.436 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:51:00.437392 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:51:00.450 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:51:00.451903 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:51:10.535 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:51:10.536207 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:51:10.548 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:51:10.549132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:51:20.660 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:51:20.660915 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:51:20.673 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:51:20.674612 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:51:30.748 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:51:30.748620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:51:30.760 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:51:30.761332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:51:40.871 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:51:40.871810 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:51:40.884 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:51:40.885219 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:51:50.995 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:51:50.996485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:51:51.011 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:51:51.011963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:52:01.081 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:52:01.081824 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:52:01.094 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:52:01.095302 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:52:11.146 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:52:11.147482 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:52:11.155 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:52:11.156737 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:52:21.255 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:52:21.255970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:52:21.268 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:52:21.269812 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:52:31.331 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:52:31.331911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:52:31.370 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:52:31.371329 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:52:41.484 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:52:41.484834 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:52:41.496 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:52:41.497420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:52:51.597 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:52:51.597725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:52:51.611 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:52:51.611923 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:53:01.716 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:53:01.717492 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:53:01.727 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:53:01.728632 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:53:11.799 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:53:11.799960 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:53:11.813 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:53:11.813445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:53:21.879 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:53:21.879857 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:53:21.891 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:53:21.891129 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:53:31.997 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:53:31.997670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:53:32.010 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:53:32.011529 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:53:42.119 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:53:42.119640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:53:42.133 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:53:42.134076 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:53:52.330 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:53:52.330622 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:53:52.343 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:53:52.343892 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:54:02.429 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:54:02.429572 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:54:02.444 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:54:02.445113 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:54:12.516 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:54:12.516698 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:54:12.529 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:54:12.529819 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:54:22.611 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:54:22.611711 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:54:22.624 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:54:22.625716 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:54:32.732 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:54:32.732574 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:54:32.743 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:54:32.744882 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:54:42.879 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:54:42.880047 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:54:42.892 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:54:42.892992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:54:52.959 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:54:52.959916 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:54:52.973 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:54:52.974421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:55:03.065 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:55:03.066227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:55:03.077 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:55:03.078564 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:55:13.165 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:55:13.165790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:55:13.178 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:55:13.179438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:55:23.343 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:55:23.344198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:55:23.354 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:55:23.355193 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:55:33.427 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:55:33.427901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:55:33.439 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:55:33.440381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:55:43.568 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:55:43.569027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:55:43.582 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:55:43.583090 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:55:53.725 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:55:53.725336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:55:53.739 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:55:53.739385 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:56:03.801 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:56:03.802635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:56:03.870 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:56:03.871350 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:56:13.976 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:56:13.977651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:56:13.988 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:56:13.989413 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:56:24.064 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:56:24.065170 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:56:24.075 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:56:24.076763 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:56:34.149 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:56:34.149947 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:56:34.159 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:56:34.160400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:56:44.230 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:56:44.231606 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:56:44.240 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:56:44.241399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:56:54.390 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:56:54.391222 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:56:54.401 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:56:54.401857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:57:04.466 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:57:04.462839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:57:04.472 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:57:04.473700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:57:14.577 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:57:14.578268 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:57:14.596 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:57:14.597306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:57:24.663 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:57:24.663905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:57:24.676 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:57:24.677569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:57:34.784 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:57:34.785348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:57:34.797 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:57:34.798754 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:57:44.873 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:57:44.874456 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:57:44.885 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:57:44.886584 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:57:54.981 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:57:54.981845 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:57:54.992 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:57:54.993684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:58:05.106 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:58:05.106513 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:58:05.119 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:58:05.119631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:58:15.217 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:58:15.218083 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:58:15.228 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:58:15.229705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:58:25.340 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:58:25.341179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:58:25.353 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:58:25.354197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:58:35.468 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:58:35.468722 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:58:35.481 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:58:35.482562 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:58:45.561 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:58:45.561871 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:58:45.577 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:58:45.577879 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:58:55.636 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:58:55.637100 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:58:55.704 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:58:55.705670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:59:05.817 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:59:05.818201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:59:05.832 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:59:05.833197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:59:15.929 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:59:15.929948 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:59:15.941 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:59:15.942204 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:59:26.062 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:59:26.062840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:59:26.075 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:59:26.075738 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:59:36.141 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:59:36.142227 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:59:36.158 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:59:36.155226 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:59:46.229 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:59:46.230433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:59:46.272 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:59:46.273311 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:59:56.389 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:59:56.390314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T01:59:56.402 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 01:59:56.403313 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:00:01.146 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:00:01.147357 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-08T02:00:01.837 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:00:01.837484 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T02:00:01.823+0000 7fb458283380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T02:00:01.838 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T02:00:01.823+0000 7fb458283380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T02:00:01.838 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T02:00:01.823+0000 7fb458283380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T02:00:01.839 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-08T02:00:06.471 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:00:06.472602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:00:06.522 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:00:06.523626 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:00:16.601 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:00:16.601336 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:00:16.614 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:00:16.615145 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:00:26.747 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:00:26.747933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:00:26.759 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:00:26.760868 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:00:36.823 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:00:36.823758 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:00:36.836 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:00:36.837469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:00:46.898 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:00:46.894789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:00:46.906 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:00:46.907599 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:00:56.966 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:00:56.967526 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:00:56.978 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:00:56.979373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:01:07.066 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:01:07.067438 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:01:07.078 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:01:07.079207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:01:17.186 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:01:17.186898 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:01:17.201 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:01:17.202070 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:01:27.286 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:01:27.287184 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:01:27.301 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:01:27.302993 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:01:37.412 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:01:37.412690 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:01:37.426 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:01:37.427160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:01:47.543 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:01:47.543662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:01:47.555 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:01:47.556231 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:01:57.664 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:01:57.664860 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:01:57.686 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:01:57.687712 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:02:07.788 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:02:07.789475 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:02:07.798 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:02:07.799800 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:02:17.914 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:02:17.914938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:02:17.926 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:02:17.927746 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:02:27.989 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:02:27.989675 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:02:28.001 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:02:28.002367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:02:38.100 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:02:38.100644 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:02:38.112 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:02:38.114033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:02:48.196 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:02:48.197059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:02:48.210 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:02:48.210954 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:02:58.289 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:02:58.289889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:02:58.301 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:02:58.302167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:03:08.398 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:03:08.398733 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:03:08.412 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:03:08.413205 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:03:18.507 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:03:18.507024 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:03:18.518 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:03:18.519283 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:03:28.592 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:03:28.593272 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:03:28.645 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:03:28.645844 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:03:38.712 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:03:38.713330 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:03:38.726 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:03:38.727356 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:03:48.789 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:03:48.790425 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:03:48.852 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:03:48.853100 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:03:58.949 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:03:58.950099 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:03:58.962 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:03:58.963195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:04:09.080 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:04:09.080958 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:04:09.091 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:04:09.091930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:04:19.213 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:04:19.214258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:04:19.222 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:04:19.223933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:04:29.363 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:04:29.364306 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:04:29.375 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:04:29.376615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:04:39.537 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:04:39.538375 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:04:39.548 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:04:39.549708 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:04:49.682 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:04:49.683512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:04:49.694 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:04:49.695791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:04:59.825 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:04:59.825764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:04:59.836 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:04:59.837034 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:05:09.989 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:05:09.990224 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:05:10.002 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:05:10.002983 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:05:20.066 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:05:20.067395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:05:20.079 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:05:20.080093 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:05:30.181 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:05:30.182393 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:05:30.228 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:05:30.229438 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:05:40.338 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:05:40.339537 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:05:40.350 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:05:40.351081 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:05:50.442 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:05:50.442765 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:05:50.452 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:05:50.452994 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:06:00.561 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:06:00.562092 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:06:00.645 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:06:00.574179 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:06:10.674 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:06:10.674680 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:06:10.687 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:06:10.687965 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:06:20.786 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:06:20.787057 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:06:20.802 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:06:20.803623 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:06:30.895 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:06:30.896724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:06:30.907 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:06:30.908508 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:06:40.992 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:06:40.993276 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:06:41.005 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:06:41.005891 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:06:51.112 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:06:51.112782 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:06:51.126 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:06:51.127082 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:07:01.203 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:07:01.204287 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:07:01.262 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:07:01.263298 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:07:11.333 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:07:11.333231 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:07:11.345 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:07:11.345683 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:07:21.464 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:07:21.465348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:07:21.476 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:07:21.477528 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:07:31.545 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:07:31.545174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:07:31.556 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:07:31.556823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:07:41.635 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:07:41.635803 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:07:41.650 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:07:41.651065 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:07:51.742 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:07:51.742963 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:07:51.754 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:07:51.755756 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:08:01.860 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:08:01.860805 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:08:01.872 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:08:01.873645 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:08:11.972 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:08:11.972653 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:08:11.983 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:08:11.984359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:08:22.077 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:08:22.078012 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:08:22.088 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:08:22.089206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:08:32.149 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:08:32.150541 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:08:32.160 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:08:32.160992 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:08:42.225 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:08:42.225464 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:08:42.237 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:08:42.237928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:08:52.389 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:08:52.390633 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:08:52.404 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:08:52.405597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:09:02.471 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:09:02.470428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:09:02.480 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:09:02.481108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:09:12.587 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:09:12.588450 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:09:12.601 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:09:12.602356 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:09:22.696 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:09:22.696840 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:09:22.708 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:09:22.708962 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:09:32.806 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:09:32.807602 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:09:32.820 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:09:32.821347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:09:42.917 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:09:42.918177 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:09:42.931 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:09:42.932182 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:09:53.033 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:09:53.034236 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:09:53.043 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:09:53.044269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:10:03.116 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:10:03.117350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:10:03.158 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:10:03.159764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:10:13.437 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:10:13.438139 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:10:13.447 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:10:13.448921 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:10:23.559 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:10:23.559555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:10:23.569 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:10:23.570428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:10:33.680 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:10:33.681483 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:10:33.693 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:10:33.694631 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:10:43.760 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:10:43.761594 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:10:43.775 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:10:43.777049 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:10:53.882 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:10:53.883201 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:10:53.894 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:10:53.895801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:11:03.998 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:11:03.999568 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:11:04.010 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:11:04.011493 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:11:14.114 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:11:14.114645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:11:14.127 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:11:14.127942 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:11:24.238 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:11:24.239115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:11:24.250 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:11:24.250957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:11:34.349 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:11:34.349898 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:11:34.361 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:11:34.362316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:11:44.487 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:11:44.487974 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:11:44.503 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:11:44.500318 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:11:54.600 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:11:54.601578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:11:54.612 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:11:54.614026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:12:04.759 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:12:04.760381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:12:04.773 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:12:04.774428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:12:14.865 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:12:14.866608 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:12:14.880 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:12:14.881530 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:12:24.947 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:12:24.948322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:12:24.959 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:12:24.960558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:12:35.056 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:12:35.056942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:12:35.068 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:12:35.069620 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:12:45.136 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:12:45.137004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:12:45.146 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:12:45.147207 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:12:55.267 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:12:55.268481 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:12:55.283 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:12:55.281720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:13:05.352 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:13:05.353280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:13:05.366 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:13:05.367317 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:13:15.501 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:13:15.500760 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:13:15.512 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:13:15.513421 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:13:25.622 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:13:25.620158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:13:25.631 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:13:25.632141 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:13:35.722 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:13:35.723514 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:13:35.735 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:13:35.736526 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:13:45.804 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:13:45.805042 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:13:45.855 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:13:45.856074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:13:55.971 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:13:55.971931 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:13:56.004 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:13:56.005704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:14:06.076 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:14:06.077649 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:14:06.090 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:14:06.091097 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:14:16.191 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:14:16.191317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:14:16.203 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:14:16.204572 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:14:26.286 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:14:26.287343 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:14:26.480 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:14:26.300771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:14:36.377 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:14:36.377657 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:14:36.420 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:14:36.421011 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:14:46.494 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:14:46.495148 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:14:46.508 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:14:46.509693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:14:56.560 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:14:56.561155 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:14:56.571 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:14:56.572505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:15:06.670 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:15:06.670749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:15:06.682 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:15:06.683367 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:15:16.785 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:15:16.785874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:15:16.798 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:15:16.799373 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:15:26.903 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:15:26.904168 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:15:26.919 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:15:26.919745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:15:36.990 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:15:36.991740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:15:37.003 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:15:37.004243 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:15:47.111 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:15:47.112320 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:15:47.124 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:15:47.125781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:15:57.214 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:15:57.215660 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:15:57.227 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:15:57.228661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:16:07.314 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:16:07.315385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:16:07.326 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:16:07.327536 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:16:17.407 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:16:17.408120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:16:17.418 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:16:17.419544 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:16:27.491 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:16:27.492120 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:16:27.505 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:16:27.506265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:16:37.613 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:16:37.614564 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:16:37.626 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:16:37.627130 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:16:41.842 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:16:41.843671 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-08T02:16:42.502 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:16:42.503189 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T02:16:42.492+0000 7fb63f718380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T02:16:42.622 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T02:16:42.492+0000 7fb63f718380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T02:16:42.623 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T02:16:42.492+0000 7fb63f718380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T02:16:42.623 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-08T02:16:47.695 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:16:47.695928 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:16:47.708 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:16:47.709485 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:16:57.771 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:16:57.772556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:16:57.798 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:16:57.798829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:17:07.920 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:17:07.921510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:17:07.934 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:17:07.935094 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:17:18.004 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:17:18.005235 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:17:18.015 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:17:18.016895 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:17:28.114 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:17:28.114742 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:17:28.127 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:17:28.128362 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:17:38.233 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:17:38.233995 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:17:38.250 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:17:38.251051 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:17:48.362 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:17:48.363032 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:17:48.371 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:17:48.372957 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:17:58.450 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:17:58.451453 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:17:58.463 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:17:58.464720 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:18:08.634 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:18:08.634998 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:18:08.645 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:18:08.647014 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:18:18.744 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:18:18.745179 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:18:18.789 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:18:18.757364 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:18:28.887 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:18:28.888050 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:18:28.904 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:18:28.905424 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:18:39.020 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:18:39.021316 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:18:39.032 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:18:39.033116 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:18:49.104 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:18:49.104882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:18:49.180 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:18:49.181429 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:18:59.257 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:18:59.258470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:18:59.269 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:18:59.270498 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:19:09.358 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:19:09.359443 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:19:09.372 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:19:09.373155 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:19:19.484 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:19:19.484937 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:19:19.497 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:19:19.498566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:19:29.563 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:19:29.564776 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:19:29.575 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:19:29.576968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:19:39.638 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:19:39.637460 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:19:39.645 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:19:39.647153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:19:49.743 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:19:49.744713 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:19:49.755 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:19:49.756938 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:19:59.860 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:19:59.861732 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:19:59.873 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:19:59.874369 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:20:09.967 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:20:09.968516 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:20:09.976 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:20:09.977996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:20:20.070 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:20:20.071719 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:20:20.141 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:20:20.142132 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:20:30.196 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:20:30.197259 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:20:30.225 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:20:30.226221 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:20:40.313 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:20:40.313933 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:20:40.328 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:20:40.329006 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:20:50.435 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:20:50.434925 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:20:50.447 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:20:50.448255 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:21:00.514 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:21:00.515178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:21:00.528 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:21:00.528931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:21:10.597 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:21:10.598433 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:21:10.610 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:21:10.611250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:21:20.748 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:21:20.748435 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:21:20.758 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:21:20.759066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:21:30.869 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:21:30.868618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:21:30.880 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:21:30.881340 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:21:40.983 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:21:40.983911 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:21:40.994 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:21:40.995985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:21:51.055 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:21:51.056180 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:21:51.066 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:21:51.067719 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:22:01.273 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:22:01.274586 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:22:01.285 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:22:01.286560 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:22:11.377 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:22:11.378772 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:22:11.390 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:22:11.391229 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:22:21.514 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:22:21.514986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:22:21.524 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:22:21.525790 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:22:31.625 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:22:31.626793 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:22:31.636 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:22:31.637731 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:22:41.739 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:22:41.739808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:22:41.750 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:22:41.752061 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:22:51.839 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:22:51.840687 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:22:51.852 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:22:51.853686 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:23:01.963 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:23:01.964478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:23:01.975 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:23:01.976359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:23:12.054 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:23:12.055471 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:23:12.114 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:23:12.115924 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:23:22.202 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:23:22.203571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:23:22.214 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:23:22.215960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:23:32.291 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:23:32.292329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:23:32.363 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:23:32.364705 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:23:42.490 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:23:42.491605 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:23:42.502 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:23:42.503386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:23:52.595 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:23:52.596299 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:23:52.603 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:23:52.605147 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:24:02.672 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:24:02.673408 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:24:02.684 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:24:02.685968 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:24:12.760 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:24:12.761239 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:24:12.819 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:24:12.820270 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:24:22.901 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:24:22.902281 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:24:22.992 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:24:22.988475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:24:33.053 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:24:33.054172 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:24:33.109 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:24:33.110493 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:24:43.181 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:24:43.181862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:24:43.190 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:24:43.192131 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:24:53.294 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:24:53.294880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:24:53.307 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:24:53.308115 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:25:03.378 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:25:03.379486 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:25:03.400 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:25:03.396981 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:25:13.467 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:25:13.467662 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:25:13.479 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:25:13.480040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:25:23.542 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:25:23.543221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:25:23.554 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:25:23.556160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:25:33.613 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:25:33.614314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:25:33.668 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:25:33.669615 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:25:43.735 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:25:43.736297 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:25:43.747 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:25:43.748232 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:25:53.861 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:25:53.861608 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:25:53.891 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:25:53.892249 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:26:04.012 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:26:04.013798 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:26:04.025 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:26:04.026688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:26:14.126 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:26:14.126247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:26:14.135 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:26:14.136677 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:26:24.210 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:26:24.211702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:26:24.220 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:26:24.222167 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:26:34.301 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:26:34.302547 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:26:34.314 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:26:34.315568 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:26:44.415 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:26:44.416967 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:26:44.426 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:26:44.427842 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:26:54.532 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:26:54.533601 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:26:54.545 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:26:54.546846 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:27:04.653 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:27:04.654174 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:27:04.665 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:27:04.667046 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:27:14.734 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:27:14.735495 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:27:14.748 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:27:14.749524 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:27:24.873 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:27:24.873808 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:27:24.883 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:27:24.884574 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:27:35.019 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:27:35.020427 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:27:35.030 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:27:35.032153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:27:45.151 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:27:45.152285 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:27:45.162 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:27:45.163766 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:27:55.235 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:27:55.235784 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:27:55.246 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:27:55.247801 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:28:05.311 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:28:05.312715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:28:05.323 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:28:05.324657 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:28:15.408 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:28:15.409571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:28:15.468 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:28:15.469328 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:28:25.546 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:28:25.547321 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:28:25.558 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:28:25.559442 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:28:35.717 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:28:35.718828 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:28:35.729 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:28:35.730688 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:28:45.818 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:28:45.818977 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:28:45.831 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:28:45.832406 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:28:55.936 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:28:55.937598 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:28:55.950 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:28:55.951707 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:29:06.049 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:29:06.050348 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:29:06.059 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:29:06.061138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:29:16.124 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:29:16.125470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:29:16.167 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:29:16.168982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:29:26.269 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:29:26.269825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:29:26.283 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:29:26.284727 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:29:36.401 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:29:36.402891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:29:36.415 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:29:36.416829 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:29:46.519 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:29:46.520666 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:29:46.532 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:29:46.533936 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:29:56.639 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:29:56.640314 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:29:56.651 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:29:56.652989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:30:06.745 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:30:06.746095 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:30:06.787 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:30:06.787843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:30:16.853 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:30:16.854018 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:30:16.940 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:30:16.941265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:30:27.038 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:30:27.039578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:30:27.050 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:30:27.051227 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:30:37.182 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:30:37.182759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:30:37.196 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:30:37.197625 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:30:47.278 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:30:47.279811 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:30:47.290 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:30:47.291655 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:30:57.390 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:30:57.391619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:30:57.405 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:30:57.406808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:31:07.507 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:31:07.508035 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:31:07.518 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:31:07.519242 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:31:17.613 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:31:17.614491 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:31:17.627 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:31:17.628505 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:31:27.722 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:31:27.723381 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:31:27.735 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:31:27.736733 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:31:37.838 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:31:37.838198 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:31:37.850 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:31:37.851381 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:31:47.919 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:31:47.920452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:31:48.151 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:31:47.931815 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:31:58.032 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:31:58.033431 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:31:58.045 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:31:58.046450 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:32:08.158 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:32:08.159176 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:32:08.170 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:32:08.171741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:32:18.242 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:32:18.243186 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:32:18.256 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:32:18.257365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:32:28.327 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:32:28.328322 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:32:28.340 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:32:28.341443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:32:38.406 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:32:38.406872 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:32:38.418 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:32:38.419668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:32:48.518 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:32:48.518997 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:32:48.527 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:32:48.528446 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:32:58.591 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:32:58.592346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:32:58.603 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:32:58.605073 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:33:08.681 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:33:08.682546 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:33:08.693 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:33:08.695160 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:33:18.814 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:33:18.815311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:33:18.829 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:33:18.830469 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:33:22.513 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:33:22.514397 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-08T02:33:23.227 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:33:23.227611 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T02:33:23.212+0000 7ff1d978a380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T02:33:23.227 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T02:33:23.212+0000 7ff1d978a380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T02:33:23.228 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T02:33:23.212+0000 7ff1d978a380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T02:33:23.228 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-08T02:33:28.932 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:33:28.933154 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:33:28.943 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:33:28.944859 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:33:39.048 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:33:39.049434 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:33:39.061 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:33:39.062610 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:33:49.138 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:33:49.139260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:33:49.196 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:33:49.197146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:33:59.262 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:33:59.263317 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:33:59.272 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:33:59.272723 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:34:09.347 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:34:09.348346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:34:09.428 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:34:09.429689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:34:19.516 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:34:19.516888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:34:19.568 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:34:19.569197 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:34:29.644 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:34:29.645663 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:34:29.702 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:34:29.703501 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:34:39.807 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:34:39.808446 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:34:39.825 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:34:39.826199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:34:49.931 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:34:49.931293 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:34:49.947 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:34:49.944285 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:35:00.036 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:35:00.036789 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:35:00.047 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:35:00.048386 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:35:10.155 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:35:10.154613 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:35:10.163 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:35:10.165052 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:35:20.224 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:35:20.224917 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:35:20.236 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:35:20.237398 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:35:30.498 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:35:30.499706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:35:30.510 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:35:30.511741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:35:40.616 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:35:40.617534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:35:40.639 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:35:40.640816 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:35:50.756 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:35:50.756980 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:35:50.770 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:35:50.771057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:36:00.900 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:36:00.901396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:36:00.922 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:36:00.916477 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:36:11.021 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:36:11.022063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:36:11.034 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:36:11.035583 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:36:21.123 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:36:21.123874 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:36:21.134 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:36:21.135057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:36:31.268 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:36:31.269465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:36:31.281 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:36:31.282902 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:36:41.359 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:36:41.360502 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:36:41.374 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:36:41.374808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:36:51.447 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:36:51.448651 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:36:51.457 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:36:51.458475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:37:01.569 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:37:01.570839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:37:01.580 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:37:01.581260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:37:11.643 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:37:11.642522 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:37:11.654 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:37:11.655138 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:37:21.746 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:37:21.747298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:37:21.762 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:37:21.763212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:37:31.823 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:37:31.823672 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:37:31.837 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:37:31.837958 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:37:41.908 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:37:41.909459 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:37:41.958 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:37:41.959483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:37:52.026 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:37:52.027575 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:37:52.039 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:37:52.040803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:38:02.116 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:38:02.117054 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:38:02.128 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:38:02.128674 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:38:12.192 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:38:12.193557 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:38:12.218 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:38:12.207035 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:38:22.321 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:38:22.321859 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:38:22.332 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:38:22.333855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:38:32.439 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:38:32.439897 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:38:32.451 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:38:32.452922 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:38:42.561 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:38:42.562098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:38:42.574 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:38:42.575807 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:38:52.685 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:38:52.686023 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:38:52.698 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:38:52.699445 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:39:02.801 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:39:02.801973 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:39:02.812 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:39:02.813931 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:39:12.919 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:39:12.920929 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:39:12.932 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:39:12.933693 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:39:23.048 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:39:23.048620 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:39:23.060 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:39:23.061960 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:39:33.171 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:39:33.172085 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:39:33.182 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:39:33.183781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:39:43.259 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:39:43.260470 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:39:43.316 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:39:43.313443 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:39:53.403 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:39:53.404048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:39:53.416 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:39:53.418119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:40:03.561 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:40:03.562631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:40:03.572 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:40:03.573853 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:40:13.659 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:40:13.660525 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:40:13.673 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:40:13.674804 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:40:23.763 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:40:23.763829 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:40:23.775 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:40:23.776359 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:40:33.869 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:40:33.870308 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:40:33.879 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:40:33.881200 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:40:43.944 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:40:43.944704 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:40:44.012 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:40:44.014106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:40:54.081 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:40:54.083035 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:40:54.092 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:40:54.093668 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:41:04.303 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:41:04.304501 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:41:04.315 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:41:04.316447 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:41:14.397 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:41:14.398794 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:41:14.407 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:41:14.409262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:41:24.507 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:41:24.507779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:41:24.518 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:41:24.520166 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:41:34.604 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:41:34.605163 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:41:34.617 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:41:34.618978 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:41:44.723 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:41:44.724269 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:41:44.735 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:41:44.736764 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:41:54.889 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:41:54.890122 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:41:54.903 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:41:54.900870 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:42:05.016 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:42:05.017579 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:42:05.028 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:42:05.029513 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:42:15.146 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:42:15.147188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:42:15.192 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:42:15.193633 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:42:25.265 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:42:25.266881 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:42:25.277 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:42:25.279468 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:42:35.341 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:42:35.342815 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:42:35.356 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:42:35.357963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:42:45.455 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:42:45.455991 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:42:45.467 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:42:45.468559 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:42:55.636 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:42:55.637554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:42:55.646 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:42:55.648279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:43:05.723 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:43:05.724552 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:43:05.796 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:43:05.797578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:43:15.893 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:43:15.894395 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:43:15.906 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:43:15.905759 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:43:26.016 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:43:26.017741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:43:26.029 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:43:26.030527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:43:36.112 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:43:36.113398 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:43:36.124 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:43:36.126176 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:43:46.199 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:43:46.200677 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:43:46.212 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:43:46.213985 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:43:56.323 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:43:56.324389 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:43:56.336 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:43:56.337342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:44:06.443 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:44:06.444350 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:44:06.455 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:44:06.457265 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:44:16.539 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:44:16.540385 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:44:16.552 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:44:16.553761 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:44:26.667 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:44:26.668074 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:44:26.679 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:44:26.680608 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:44:36.802 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:44:36.802852 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:44:36.813 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:44:36.814578 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:44:46.931 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:44:46.931862 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:44:46.942 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:44:46.943828 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:44:57.048 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:44:57.048578 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:44:57.061 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:44:57.062706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:45:07.135 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:45:07.136679 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:45:07.146 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:45:07.147643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:45:17.261 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:45:17.262138 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:45:17.274 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:45:17.275190 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:45:27.405 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:45:27.406891 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:45:27.418 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:45:27.420327 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:45:37.515 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:45:37.516183 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:45:37.528 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:45:37.529970 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:45:47.616 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:45:47.617372 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:45:47.628 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:45:47.630122 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:45:57.698 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:45:57.699101 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:45:57.711 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:45:57.712814 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:46:07.781 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:46:07.782110 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:46:07.794 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:46:07.795909 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:46:17.908 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:46:17.909846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:46:17.923 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:46:17.924531 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:46:27.996 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:46:27.997137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:46:28.054 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:46:28.054375 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:46:38.119 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:46:38.120821 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:46:38.130 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:46:38.131911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:46:48.205 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:46:48.206311 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:46:48.218 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:46:48.219785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:46:58.303 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:46:58.304689 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:46:58.316 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:46:58.317745 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:47:08.427 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:47:08.429108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:47:08.440 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:47:08.441597 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:47:18.540 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:47:18.540943 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:47:18.553 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:47:18.553928 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:47:28.680 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:47:28.681754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:47:28.694 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:47:28.695967 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:47:38.809 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:47:38.809880 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:47:38.822 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:47:38.823913 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:47:48.949 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:47:48.950275 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:47:48.962 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:47:48.963387 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:47:59.038 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:47:59.036640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:47:59.048 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:47:59.050057 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:48:09.122 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:48:09.122938 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:48:09.133 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:48:09.135114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:48:19.316 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:48:19.317098 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:48:19.328 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:48:19.329956 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:48:29.413 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:48:29.413006 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:48:29.425 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:48:29.427365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:48:39.493 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:48:39.494352 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:48:39.504 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:48:39.505404 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:48:49.579 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:48:49.580616 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:48:49.593 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:48:49.594576 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:48:59.693 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:48:59.694131 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:48:59.705 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:48:59.706899 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:49:09.811 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:49:09.812465 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:49:09.827 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:49:09.825079 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:49:19.898 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:49:19.899773 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:49:19.955 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:49:19.956871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:49:30.037 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:49:30.034048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:49:30.054 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:49:30.055948 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:49:40.128 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:49:40.124837 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:49:40.138 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:49:40.139269 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:49:50.360 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:49:50.362002 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:49:50.374 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:49:50.375643 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:50:00.432 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:50:00.433478 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:50:00.446 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:50:00.447575 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:50:03.232 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:50:03.233614 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-08T02:50:03.878 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:50:03.879466 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T02:50:03.868+0000 7f70bf149380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T02:50:03.879 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T02:50:03.868+0000 7f70bf149380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T02:50:03.879 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T02:50:03.868+0000 7f70bf149380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T02:50:03.879 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-08T02:50:10.566 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:50:10.567048 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:50:10.575 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:50:10.576836 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:50:20.683 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:50:20.684735 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:50:20.696 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:50:20.698084 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:50:30.816 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:50:30.817833 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:50:30.831 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:50:30.832400 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:50:40.944 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:50:40.945474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:50:40.956 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:50:40.958102 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:50:51.083 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:50:51.083842 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:50:51.116 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:50:51.096008 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:51:01.169 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:51:01.170736 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:51:01.179 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:51:01.180982 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:51:11.342 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:51:11.343124 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:51:11.356 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:51:11.357824 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:51:21.460 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:51:21.461105 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:51:21.474 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:51:21.475234 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:51:31.586 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:51:31.586144 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:51:31.614 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:51:31.598312 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:51:41.705 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:51:41.706870 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:51:41.719 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:51:41.720692 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:51:51.799 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:51:51.800654 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:51:51.809 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:51:51.810361 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:52:01.893 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:52:01.894411 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:52:01.903 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:52:01.905266 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:52:11.972 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:52:11.973364 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:52:11.986 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:52:11.986403 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:52:22.091 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:52:22.092737 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:52:22.100 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:52:22.102133 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:52:32.208 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:52:32.208966 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:52:32.220 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:52:32.222064 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:52:42.292 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:52:42.293936 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:52:42.304 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:52:42.306208 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:52:52.402 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:52:52.403774 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:52:52.415 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:52:52.416021 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:53:02.486 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:53:02.487093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:53:02.502 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:53:02.503206 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:53:12.600 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:53:12.601072 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:53:12.617 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:53:12.618642 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:53:22.719 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:53:22.720298 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:53:22.730 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:53:22.731980 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:53:32.799 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:53:32.800494 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:53:32.815 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:53:32.813833 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:53:42.886 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:53:42.887188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:53:42.901 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:53:42.902566 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:53:52.972 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:53:52.973097 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:53:52.985 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:53:52.986519 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:54:03.056 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:54:03.057869 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:54:03.069 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:54:03.070871 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:54:13.144 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:54:13.145457 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:54:13.160 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:54:13.161414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:54:23.257 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:54:23.258864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:54:23.269 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:54:23.270798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:54:33.337 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:54:33.335055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:54:33.351 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:54:33.350595 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:54:43.445 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:54:43.445641 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:54:43.456 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:54:43.457661 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:54:53.576 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:54:53.577508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:54:53.589 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:54:53.589346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:55:03.843 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:55:03.844807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:55:03.856 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:55:03.858043 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:55:13.957 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:55:13.958627 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:55:13.971 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:55:13.973076 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:55:24.065 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:55:24.066468 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:55:24.077 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:55:24.078397 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:55:34.160 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:55:34.161780 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:55:34.171 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:55:34.172925 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:55:44.245 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:55:44.246725 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:55:44.256 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:55:44.257339 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:55:54.355 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:55:54.356640 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:55:54.367 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:55:54.368614 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:56:04.430 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:56:04.431589 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:56:04.443 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:56:04.444933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:56:14.547 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:56:14.547838 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:56:14.574 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:56:14.561055 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:56:24.669 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:56:24.670978 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:56:24.680 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:56:24.682153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:56:34.783 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:56:34.784561 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:56:34.795 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:56:34.796040 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:56:44.900 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:56:44.900889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:56:44.911 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:56:44.912316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:56:54.986 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:56:54.987535 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:56:54.997 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:56:54.998183 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:57:05.056 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:57:05.057158 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:57:05.068 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:57:05.070253 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:57:15.149 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:57:15.150841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:57:15.162 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:57:15.163734 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:57:25.270 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:57:25.271034 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:57:25.285 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:57:25.286389 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:57:35.382 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:57:35.383779 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:57:35.393 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:57:35.394821 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:57:45.475 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:57:45.475696 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:57:45.487 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:57:45.489025 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:57:55.557 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:57:55.558353 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:57:55.572 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:57:55.573345 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:58:05.694 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:58:05.695534 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:58:05.708 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:58:05.709135 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:58:15.790 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:58:15.791331 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:58:15.804 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:58:15.805613 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:58:25.887 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:58:25.887670 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:58:25.943 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:58:25.944354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:58:36.050 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:58:36.051351 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:58:36.064 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:58:36.065262 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:58:46.140 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:58:46.141769 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:58:46.152 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:58:46.153933 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:58:56.220 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:58:56.221646 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:58:56.298 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:58:56.299832 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:59:06.440 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:59:06.441108 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:59:06.466 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:59:06.467390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:59:16.551 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:59:16.552512 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:59:16.564 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:59:16.565839 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:59:26.668 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:59:26.668658 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:59:26.686 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:59:26.683863 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:59:36.742 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:59:36.742082 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:59:36.753 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:59:36.754347 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:59:46.818 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:59:46.819889 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:59:46.831 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:59:46.832366 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:59:56.936 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:59:56.937368 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T02:59:56.952 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 02:59:56.953670 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:00:07.047 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:00:07.047970 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:00:07.060 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:00:07.061675 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:00:17.148 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:00:17.148985 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:00:17.160 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:00:17.161565 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:00:27.252 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:00:27.253190 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:00:27.263 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:00:27.264452 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:00:37.330 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:00:37.331152 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:00:37.593 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:00:37.343110 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:00:47.438 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:00:47.439247 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:00:47.450 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:00:47.451640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:00:57.540 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:00:57.541205 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:00:57.553 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:00:57.552627 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:01:07.621 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:01:07.622790 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:01:07.635 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:01:07.636963 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:01:17.714 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:01:17.715609 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:01:17.726 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:01:17.727896 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:01:27.833 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:01:27.834717 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:01:27.847 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:01:27.848277 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:01:37.995 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:01:37.996635 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:01:38.008 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:01:38.009399 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:01:48.218 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:01:48.218715 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:01:48.232 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:01:48.233794 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:01:58.320 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:01:58.320927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:01:58.331 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:01:58.332173 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:02:08.434 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:02:08.434802 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:02:08.445 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:02:08.446905 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:02:18.529 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:02:18.530611 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:02:18.580 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:02:18.581883 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:02:28.650 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:02:28.651305 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:02:28.662 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:02:28.663346 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:02:38.731 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:02:38.732839 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:02:38.745 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:02:38.746309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:02:48.848 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:02:48.849574 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:02:48.860 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:02:48.862146 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:02:58.968 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:02:58.968909 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:02:58.979 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:02:58.981051 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:03:09.096 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:03:09.097309 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:03:09.118 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:03:09.116217 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:03:19.233 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:03:19.234261 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:03:19.245 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:03:19.246823 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:03:29.347 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:03:29.348508 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:03:29.358 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:03:29.359918 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:03:39.484 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:03:39.485485 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:03:39.495 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:03:39.497083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:03:49.615 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:03:49.615788 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:03:49.627 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:03:49.628862 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:03:59.713 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:03:59.714087 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:03:59.723 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:03:59.725083 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:04:09.792 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:04:09.793618 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:04:09.805 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:04:09.806724 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:04:19.881 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:04:19.881971 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:04:19.893 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:04:19.894458 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:04:29.970 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:04:29.971013 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:04:29.980 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:04:29.982091 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:04:40.042 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:04:40.043225 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:04:40.053 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:04:40.055099 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:04:50.174 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:04:50.174921 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:04:50.186 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:04:50.187527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:05:00.268 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:05:00.269515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:05:00.283 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:05:00.284741 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:05:10.350 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:05:10.345864 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:05:10.358 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:05:10.359295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:05:20.423 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:05:20.423739 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:05:20.489 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:05:20.490276 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:05:30.603 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:05:30.604061 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:05:30.612 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:05:30.614144 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:05:40.740 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:05:40.741661 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:05:40.754 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:05:40.755434 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:05:50.848 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:05:50.848846 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:05:50.861 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:05:50.862433 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:06:00.969 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:06:00.970279 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:06:00.980 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:06:00.980785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:06:11.093 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:06:11.094699 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:06:11.108 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:06:11.109420 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:06:21.208 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:06:21.209668 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:06:21.221 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:06:21.222435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:06:31.292 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:06:31.292945 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:06:31.303 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:06:31.304048 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:06:41.413 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:06:41.414419 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:06:41.426 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:06:41.427212 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:06:43.887 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:06:43.886504 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-08T03:06:44.531 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:06:44.530905 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T03:06:44.519+0000 7f50ca653380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T03:06:44.532 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T03:06:44.519+0000 7f50ca653380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T03:06:44.532 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T03:06:44.519+0000 7f50ca653380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T03:06:44.532 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-08T03:06:51.528 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:06:51.529691 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:06:51.556 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:06:51.557937 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:07:01.628 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:07:01.629007 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:07:01.649 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:07:01.642394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:07:11.741 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:07:11.741730 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:07:11.753 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:07:11.754798 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:07:21.849 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:07:21.850740 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:07:21.860 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:07:21.862127 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:07:31.960 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:07:31.961329 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:07:31.971 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:07:31.972771 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:07:42.053 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:07:42.053901 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:07:42.061 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:07:42.062775 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:07:52.193 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:07:52.194370 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:07:52.206 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:07:52.207838 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:08:02.265 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:08:02.266109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:08:02.311 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:08:02.311267 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:08:12.434 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:08:12.435009 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:08:12.448 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:08:12.449512 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:08:22.550 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:08:22.551335 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:08:22.563 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:08:22.564228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:08:32.738 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:08:32.739521 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:08:32.748 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:08:32.749889 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:08:42.881 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:08:42.882868 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:08:42.896 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:08:42.897989 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:08:52.995 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:08:52.995749 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:08:53.005 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:08:53.006319 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:09:03.077 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:09:03.078510 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:09:03.086 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:09:03.088074 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:09:13.180 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:09:13.181178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:09:13.215 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:09:13.191976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:09:23.291 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:09:23.291989 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:09:23.306 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:09:23.306906 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:09:33.375 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:09:33.376188 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:09:33.386 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:09:33.387322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:09:43.486 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:09:43.486986 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:09:43.501 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:09:43.502444 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:09:53.567 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:09:53.567884 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:09:53.577 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:09:53.578594 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:10:03.675 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:10:03.675827 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:10:03.687 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:10:03.687997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:10:13.777 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:10:13.777807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:10:13.789 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:10:13.790377 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:10:23.927 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:10:23.928229 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:10:23.957 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:10:23.958558 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:10:34.095 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:10:34.095250 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:10:34.108 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:10:34.110053 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:10:44.179 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:10:44.179825 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:10:44.193 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:10:44.193106 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:10:54.303 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:10:54.303801 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:10:54.315 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:10:54.316136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:11:04.414 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:11:04.415204 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:11:04.430 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:11:04.431033 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:11:14.544 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:11:14.544814 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:11:14.559 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:11:14.560063 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:11:24.623 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:11:24.624063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:11:24.636 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:11:24.637546 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:11:34.706 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:11:34.707104 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:11:34.771 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:11:34.772213 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:11:44.849 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:11:44.850041 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:11:44.863 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:11:44.863781 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:11:54.969 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:11:54.969448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:11:54.982 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:11:54.983088 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:12:05.097 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:12:05.098221 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:12:05.107 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:12:05.108647 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:12:15.233 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:12:15.233754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:12:15.241 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:12:15.242747 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:12:25.313 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:12:25.313358 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:12:25.325 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:12:25.325753 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:12:35.425 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:12:35.425792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:12:35.437 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:12:35.437940 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:12:45.552 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:12:45.552759 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:12:45.568 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:12:45.568428 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:12:55.660 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:12:55.660705 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:12:55.673 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:12:55.673717 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:13:05.797 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:13:05.797258 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:13:05.827 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:13:05.809067 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:13:15.893 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:13:15.893703 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:13:15.943 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:13:15.944015 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:13:26.043 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:13:26.043472 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:13:26.075 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:13:26.075769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:13:36.151 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:13:36.151724 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:13:36.222 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:13:36.222666 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:13:46.354 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:13:46.354554 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:13:46.372 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:13:46.369005 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:13:56.531 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:13:56.531452 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:13:56.546 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:13:56.546894 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:14:06.646 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:14:06.646428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:14:06.659 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:14:06.660068 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:14:16.728 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:14:16.727942 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:14:16.741 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:14:16.741857 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:14:26.809 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:14:26.809764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:14:26.869 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:14:26.869552 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:14:37.002 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:14:37.002045 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:14:37.014 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:14:37.014592 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:14:47.090 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:14:47.090093 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:14:47.159 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:14:47.159390 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:14:57.264 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:14:57.263723 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:14:57.277 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:14:57.277260 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:15:07.409 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:15:07.409160 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:15:07.421 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:15:07.421543 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:15:17.568 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:15:17.568260 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:15:17.581 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:15:17.581791 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:15:27.678 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:15:27.678399 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:15:27.691 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:15:27.691527 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:15:37.772 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:15:37.771729 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:15:37.787 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:15:37.786865 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:15:47.865 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:15:47.865423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:15:47.883 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:15:47.883306 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:15:57.951 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:15:57.951378 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:15:57.963 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:15:57.963545 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:16:08.029 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:16:08.029178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:16:08.042 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:16:08.042322 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:16:18.112 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:16:18.112209 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:16:18.126 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:16:18.126279 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:16:28.277 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:16:28.276792 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:16:28.310 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:16:28.309997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:16:38.377 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:16:38.376910 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:16:38.387 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:16:38.387332 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:16:48.458 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:16:48.457720 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:16:48.473 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:16:48.472977 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:16:58.548 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:16:58.547841 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:16:58.561 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:16:58.560973 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:17:08.627 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:17:08.626461 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:17:08.640 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:17:08.640120 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:17:18.705 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:17:18.705115 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:17:18.717 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:17:18.716904 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:17:28.799 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:17:28.798831 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:17:28.872 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:17:28.870930 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:17:38.954 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:17:38.953011 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:17:38.966 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:17:38.965662 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:17:49.039 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:17:49.038232 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:17:49.100 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:17:49.099784 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:17:59.197 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:17:59.196383 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:17:59.210 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:17:59.209689 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:18:09.314 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:18:09.313775 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:18:09.328 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:18:09.327718 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:18:19.408 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:18:19.408346 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:18:19.423 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:18:19.422475 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:18:29.492 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:18:29.491396 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:18:29.504 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:18:29.504354 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:18:39.603 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:18:39.602004 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:18:39.614 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:18:39.613640 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:18:49.708 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:18:49.708063 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:18:49.721 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:18:49.719999 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:18:59.828 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:18:59.827555 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:18:59.837 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:18:59.836997 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:19:09.908 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:19:09.907345 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:19:09.922 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:19:09.921372 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:19:19.981 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:19:19.980631 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:19:19.991 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:19:19.990414 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:19:30.123 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:19:30.119893 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:19:30.134 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:19:30.132363 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:19:40.244 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:19:40.243694 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:19:40.257 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:19:40.257195 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:19:50.340 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:19:50.339746 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:19:50.408 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:19:50.407308 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:20:00.526 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:20:00.525967 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:20:00.543 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:20:00.543045 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:20:10.661 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:20:10.660971 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:20:10.675 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:20:10.674700 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:20:20.767 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:20:20.764159 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:20:20.776 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:20:20.775175 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:20:30.905 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:20:30.904927 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:20:30.921 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:20:30.918030 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:20:40.986 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:20:40.985676 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:20:41.042 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:20:41.041991 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:20:51.161 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:20:51.160645 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:20:51.174 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:20:51.173245 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:21:01.263 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:21:01.262280 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:21:01.277 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:21:01.276374 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:21:11.340 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:21:11.339643 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:21:11.356 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:21:11.352310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:21:21.426 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:21:21.425129 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:21:21.438 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:21:21.438228 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:21:31.518 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:21:31.516984 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:21:31.531 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:21:31.530199 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:21:41.632 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:21:41.631474 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:21:41.644 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:21:41.643855 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:21:51.742 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:21:51.740830 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:21:51.754 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:21:51.753316 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:22:01.875 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:22:01.874178 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:22:01.888 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:22:01.887471 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:22:11.982 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:22:11.981423 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:22:11.995 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:22:11.994743 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:22:22.202 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:22:22.201764 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:22:22.218 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:22:22.217026 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:22:32.345 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:22:32.344338 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:22:32.359 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:22:32.358484 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:22:42.433 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:22:42.431882 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:22:42.445 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:22:42.443803 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:22:52.517 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:22:52.516600 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:22:52.530 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:22:52.529664 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:23:02.606 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:23:02.605365 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:23:02.619 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:23:02.618119 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:23:12.743 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:23:12.742407 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:23:12.755 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:23:12.754739 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:23:22.847 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:23:22.846628 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:23:22.862 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:23:22.861769 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:23:24.538 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:23:24.537238 I | op-mon: parsing mon endpoints: a=10.105.146.196:6789,b=10.98.222.16:6789,c=10.96.111.213:6789 2022-03-08T03:23:25.208 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:23:25.206474 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/foo". failed to check for object buckets. failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "foo": failed to create s3 user. 2022-03-08T03:23:25.179+0000 7fae2a40c380 0 failed reading zonegroup info: ret -2 (2) No such file or directory 2022-03-08T03:23:25.208 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T03:23:25.179+0000 7fae2a40c380 0 ERROR: failed to start notify service ((2) No such file or directory 2022-03-08T03:23:25.208 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08T03:23:25.179+0000 7fae2a40c380 0 ERROR: failed to init services (ret=(2) No such file or directory) 2022-03-08T03:23:25.209 INFO:tasks.rook.operator.smithi078.stdout:couldn't init storage provider. : exit status 5 2022-03-08T03:23:32.931 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:23:32.930489 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:23:32.942 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:23:32.941483 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:23:43.041 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:23:43.039807 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:23:43.051 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:23:43.050547 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:23:53.120 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:23:53.119786 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:23:53.132 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:23:53.131690 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:24:03.215 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:24:03.214428 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:24:03.263 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:24:03.262217 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:24:13.362 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:24:13.360439 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:24:13.376 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:24:13.375704 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:24:23.481 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:24:23.479888 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:24:23.494 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:24:23.493843 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:24:33.578 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:24:33.577077 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:24:33.590 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:24:33.589360 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:24:43.697 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:24:43.696375 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:24:43.710 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:24:43.708757 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:24:53.794 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:24:53.792741 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:24:53.860 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:24:53.859365 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:25:03.927 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:25:03.925822 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:25:03.991 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:25:03.990408 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:25:14.105 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:25:14.104454 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:25:14.118 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:25:14.117024 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:25:24.245 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:25:24.244334 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:25:24.261 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:25:24.260066 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:25:34.364 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:25:34.363055 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:25:34.375 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:25:34.374342 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:25:44.488 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:25:44.486849 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:25:44.499 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:25:44.498808 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:25:54.565 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:25:54.564237 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:25:54.580 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:25:54.579126 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:26:04.705 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:26:04.703556 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:26:04.721 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:26:04.719951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:26:14.788 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:26:14.786702 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:26:14.801 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:26:14.800309 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:26:24.864 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:26:24.863195 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:26:24.929 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:26:24.928114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:26:34.989 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:26:34.987614 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:26:35.063 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:26:35.060602 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:26:45.127 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:26:45.126427 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:26:45.141 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:26:45.140194 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:26:55.204 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:26:55.203240 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:26:55.218 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:26:55.217109 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:27:05.288 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:27:05.287217 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:27:05.302 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:27:05.300976 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:27:15.375 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:27:15.374075 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:27:15.388 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:27:15.386910 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:27:25.468 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:27:25.466496 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:27:25.516 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:27:25.514996 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:27:35.588 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:27:35.586449 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:27:35.600 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:27:35.599706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:27:45.673 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:27:45.672220 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:27:45.732 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:27:45.730487 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:27:55.857 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:27:55.855878 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:27:55.869 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:27:55.867869 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:28:05.934 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:28:05.932068 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:28:05.950 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:28:05.949295 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:28:16.068 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:28:16.066515 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:28:16.080 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:28:16.078825 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:28:26.155 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:28:26.153279 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:28:26.167 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:28:26.166320 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:28:36.244 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:28:36.242580 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:28:36.255 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:28:36.254153 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:28:46.335 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:28:46.333309 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:28:46.350 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:28:46.347580 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:28:56.505 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:28:56.503137 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:28:56.516 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:28:56.514706 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:29:06.586 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:29:06.585059 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:29:06.598 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:29:06.597188 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:29:16.711 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:29:16.709934 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:29:16.721 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:29:16.719785 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:29:26.818 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:29:26.817071 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:29:26.859 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:29:26.857541 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:29:36.982 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:29:36.980865 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:29:36.993 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:29:36.992150 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:29:47.087 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:29:47.086192 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:29:47.154 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:29:47.153250 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:29:57.261 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:29:57.260027 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:29:57.274 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:29:57.273310 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:30:07.433 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:30:07.431727 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:30:07.509 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:30:07.508114 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:30:17.626 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:30:17.624923 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:30:17.637 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:30:17.636136 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:30:27.739 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:30:27.737379 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:30:27.750 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:30:27.748911 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:30:37.826 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:30:37.824835 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:30:37.840 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:30:37.838616 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:30:47.983 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:30:47.981754 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:30:47.995 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:30:47.994435 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:30:58.095 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:30:58.093571 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:30:58.107 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:30:58.106051 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:31:08.219 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:31:08.217706 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:31:08.233 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:31:08.231900 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:31:18.362 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:31:18.360384 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:31:18.373 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:31:18.372244 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:31:28.435 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:31:28.433272 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:31:28.486 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:31:28.484108 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:31:38.590 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:31:38.588128 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:31:38.603 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:31:38.601979 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:31:48.672 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:31:48.670319 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:31:48.685 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:31:48.683394 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:31:58.750 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:31:58.748976 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:31:58.763 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:31:58.762287 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:32:08.876 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:32:08.874905 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:32:08.887 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:32:08.885951 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:32:18.956 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:32:18.954448 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:32:18.969 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:32:18.967569 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:32:29.084 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:32:29.082109 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:32:29.097 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:32:29.095358 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:32:39.159 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:32:39.157619 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T03:32:39.216 INFO:tasks.rook.operator.smithi078.stdout:2022-03-08 03:32:39.214684 E | ceph-cluster-controller: failed to reconcile CephCluster "rook-ceph/rook-ceph". CephCluster "rook-ceph/rook-ceph" will not be deleted until all dependents are removed: CephObjectStore: [foo] 2022-03-08T05:53:37.012 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-03-08T05:53:37.042 DEBUG:teuthology.task.console_log:Killing console logger for smithi078 2022-03-08T05:53:37.044 DEBUG:teuthology.task.console_log:Killing console logger for smithi166 2022-03-08T05:53:37.044 DEBUG:teuthology.task.console_log:Killing console logger for smithi170 2022-03-08T05:53:37.044 DEBUG:teuthology.exit:Finished running handlers