2022-08-05T15:49:34.545 INFO:root:teuthology version: 1.1.1.dev407+g9e4bf63f 2022-08-05T15:49:34.554 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2022-08-05_14:49:51-rados-wip-yuri4-testing-2022-08-04-1413-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2022-08-05_14:49:51-rados-wip-yuri4-testing-2022-08-04-1413-distro-default-smithi/6959556 --owner scheduled_yuriw@teuthology --verbose --description rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/3-node k8s/1.21 net/flannel rook/master} -- /home/teuthworker/archive/yuriw-2022-08-05_14:49:51-rados-wip-yuri4-testing-2022-08-04-1413-distro-default-smithi/6959556/orig.config.yaml 2022-08-05T15:49:34.681 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-08-05T15:49:34.767 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-08-05_14:49:51-rados-wip-yuri4-testing-2022-08-04-1413-distro-default-smithi/6959556 branch: wip-yuri4-testing-2022-08-04-1413 description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/3-node k8s/1.21 net/flannel rook/master} email: ceph-qa@ceph.io first_in_suite: false job_id: '6959556' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2022-08-05_14:49:51-rados-wip-yuri4-testing-2022-08-04-1413-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri4-testing-2022-08-04-1413 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 4a7e6c4ac29de794744a2ce76ca2b36bc637894d ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 4a7e6c4ac29de794744a2ce76ca2b36bc637894d kernel: hwe: true kubeadm: pod_network: flannel version: '1.21' rook: rook_image: rook/ceph:master workunit: branch: wip-yuri4-testing-2022-08-04-1413 sha1: 4a7e6c4ac29de794744a2ce76ca2b36bc637894d owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.a - - host.b - client.b - - host.c - client.c sha1: 4a7e6c4ac29de794744a2ce76ca2b36bc637894d sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri4-testing-2022-08-04-1413 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_4a7e6c4ac29de794744a2ce76ca2b36bc637894d/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 4a7e6c4ac29de794744a2ce76ca2b36bc637894d targets: smithi059.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwea/Iirc1kzi7g4++3ZYg8ZzcV3HAIL0a2QPGSzHOXLlNjv5Aw83/K2S++7Oc5xnaKjfiT88I56xzTKbZ3NOTPMQrz7hFnSgzObbILWZJnlngTCePPrD0dLn/MEjXVTx/WaVCawO5x1ygswCcEjV3OUOwx8Z9DqL2LbcfYd4ZseDczLnpQoBIktBNiCtG7kvSLVLNXRtzAa2xIos3boSwTciOppVt4O9CKPseDAPF3U+c2fOWz/Vf2tV39ahhaDGPSv2wQv08YbLFq7M8fSdxNSFKzf8Suaipyz3DFYsjkDmxapmHnOltbr0dyM7X37sVPF/nBgvS+tjwUZJo5qyrk2tg/GFlkZOwPURUqlxKFq0udDQdnyPQvjpKh2mExpLi6rN+do2OMxgSFxPmbxP6RrRlUjEyks56u/oQ2rolA/znTrG5wXE0kq6znzboz3a/9pBtBRQGiWpcoz+mGEySC6o2+A+LrAT0I5OX0kDepZoKvoaGE3L0Ez0NayJ8n3M= smithi157.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCnVoKVrn3sZuiTXZEQz6r73J9dDTIqiTFapvnv8+CciqE+0IBzmcqTH62uzOJPJ9UFBigiWm0paY33kJ7QSu5ddaqPT5joAtxhFg8IIlM1lvqYjbDbgaxBC6/8tsxRppsWYWpbP3eyBEQWanpat7OLk6cr+zIivzEHneQbmBdsEtjb/ii8SlB/SSTRoI/e52/TUM9uLunIReCoyWDwZpVxerz+v1DHO96VMYgtxx71v9vs2HeNJaJ62Bz5dRo3eWpfQ9Rbd0A0KcGZmbhov6TLQc8ycF2rn8388ahgpxl0ZgUllQdJiZ+bLHfLtGNgV3xOSaceu4LPDG6qkMKafXb82AT3z3O1PraHxfg4syHFBbfEXpHm8XwD3OlSJzJ0X5zc6sYBaCfmu3QOhtnNT+K9QdPVkV2iuQN+pzUXv/pMmUMPmCjLH76FgtaxAwNOczusoRF9qGR3jcxCVbsCc//2/f7ZoPrqnWW3r7CilBibEtvHSl1vaA2UxnB6hOBvCL8= smithi201.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6BWqMWMl/8HUTPMu35Ryd9ydQYtDXEofXrP4adLcwMgGoIbCjT0I4/XVJ8Z6EGysUqGnzFq0EdDuhWIsLd5w16gKVg+zo0LqjFGWMIv2fEu9CRCqZZfesyGNJZ0jJlpRcwXFG4J/5eppES3gfrCZ5Dz4hgHzavS7ST3zk1tSATPp111t/nzWdPaIWGjVVujUIRQEM4hCqe/qow9DXUwmRhDNqcoAU6Wz0DYplsR4eyoDBBWSN++Ro6BCNFmOGR2HhdT0d0sCFAzMJNjLRQQoEOOrQRBh9TwKlGdkqdazWVH591mOLBIY8CzyyWrVFZwrWYNkjt3s7H+TsQasHjfuv8shbJPwLvimqsiKphMc14ca5durzw0lTERRXnaR75huj3CzoYGsdnzeXya4lWzAlxtXV0ZMTz7nnAOSgf+Pl6k4S+jJEpKIKGvTskakR7dWDUEz+b0hKzXEv0ihSE46dJXjwoFjTljzWXWZzvkoeP1apJ0mjTvC5WPf2eo10sYU= tasks: - kubeadm: null - nvme_loop: null - rook: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9e4bf63f00ab3f82a26635c2779f4c3f1b73fb53 timestamp: 2022-08-05_14:49:51 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1713 2022-08-05T15:49:34.768 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_4a7e6c4ac29de794744a2ce76ca2b36bc637894d/qa; will attempt to use it 2022-08-05T15:49:34.769 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_4a7e6c4ac29de794744a2ce76ca2b36bc637894d/qa/tasks 2022-08-05T15:49:34.769 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-08-05T15:49:34.772 INFO:teuthology.task.internal:Checking packages... 2022-08-05T15:49:34.838 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '4a7e6c4ac29de794744a2ce76ca2b36bc637894d' 2022-08-05T15:49:34.840 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-08-05T15:49:34.841 INFO:teuthology.packaging:ref: None 2022-08-05T15:49:34.841 INFO:teuthology.packaging:tag: None 2022-08-05T15:49:34.842 INFO:teuthology.packaging:branch: wip-yuri4-testing-2022-08-04-1413 2022-08-05T15:49:34.842 INFO:teuthology.packaging:sha1: 4a7e6c4ac29de794744a2ce76ca2b36bc637894d 2022-08-05T15:49:34.851 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri4-testing-2022-08-04-1413 2022-08-05T15:49:35.059 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-14045-g4a7e6c4a-1focal 2022-08-05T15:49:35.060 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-08-05T15:49:35.071 INFO:teuthology.task.internal:no buildpackages task found 2022-08-05T15:49:35.072 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-08-05T15:49:35.119 INFO:teuthology.task.internal:Saving configuration 2022-08-05T15:49:35.154 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-08-05T15:49:35.172 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-08-05T15:49:35.205 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-08-05_14:49:51-rados-wip-yuri4-testing-2022-08-04-1413-distro-default-smithi/6959556', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-08-05 15:43:03.313479', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILEnjzwA3KCsp/rsdAMKUd4NcehnN0aABIvLnEtNMRDD'} 2022-08-05T15:49:35.246 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-08-05_14:49:51-rados-wip-yuri4-testing-2022-08-04-1413-distro-default-smithi/6959556', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-08-05 15:43:03.314986', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGpWx2Y67VnOCvLxRAM0fv/ilhUVsfeBpCxnk6P1HP59'} 2022-08-05T15:49:35.290 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi201.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-08-05_14:49:51-rados-wip-yuri4-testing-2022-08-04-1413-distro-default-smithi/6959556', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-08-05 15:43:03.316038', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMImJRtWqWy7ZGgLst2jV/3fEiB00aRX1C0FRk2rBsLC'} 2022-08-05T15:49:35.291 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-08-05T15:49:35.304 INFO:teuthology.task.internal:roles: ubuntu@smithi059.front.sepia.ceph.com - ['host.a', 'client.a'] 2022-08-05T15:49:35.305 INFO:teuthology.task.internal:roles: ubuntu@smithi157.front.sepia.ceph.com - ['host.b', 'client.b'] 2022-08-05T15:49:35.306 INFO:teuthology.task.internal:roles: ubuntu@smithi201.front.sepia.ceph.com - ['host.c', 'client.c'] 2022-08-05T15:49:35.306 INFO:teuthology.run_tasks:Running task console_log... 2022-08-05T15:49:35.759 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7febb693fb70>, signals=[15]) 2022-08-05T15:49:35.761 INFO:teuthology.run_tasks:Running task internal.connect... 2022-08-05T15:49:35.789 INFO:teuthology.task.internal:Opening connections... 2022-08-05T15:49:35.795 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi059.front.sepia.ceph.com 2022-08-05T15:49:35.801 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-08-05T15:49:36.086 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi157.front.sepia.ceph.com 2022-08-05T15:49:36.087 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-08-05T15:49:36.279 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi201.front.sepia.ceph.com 2022-08-05T15:49:36.281 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi201.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-08-05T15:49:36.555 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-08-05T15:49:36.564 DEBUG:teuthology.orchestra.run.smithi059:> uname -m 2022-08-05T15:49:36.587 INFO:teuthology.orchestra.run.smithi059.stdout:x86_64 2022-08-05T15:49:36.588 DEBUG:teuthology.orchestra.run.smithi059:> cat /etc/os-release 2022-08-05T15:49:36.640 INFO:teuthology.orchestra.run.smithi059.stdout:NAME="Ubuntu" 2022-08-05T15:49:36.640 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-08-05T15:49:36.641 INFO:teuthology.orchestra.run.smithi059.stdout:ID=ubuntu 2022-08-05T15:49:36.641 INFO:teuthology.orchestra.run.smithi059.stdout:ID_LIKE=debian 2022-08-05T15:49:36.641 INFO:teuthology.orchestra.run.smithi059.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-08-05T15:49:36.642 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_ID="20.04" 2022-08-05T15:49:36.643 INFO:teuthology.orchestra.run.smithi059.stdout:HOME_URL="https://www.ubuntu.com/" 2022-08-05T15:49:36.644 INFO:teuthology.orchestra.run.smithi059.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-08-05T15:49:36.644 INFO:teuthology.orchestra.run.smithi059.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-08-05T15:49:36.645 INFO:teuthology.orchestra.run.smithi059.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-08-05T15:49:36.645 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_CODENAME=focal 2022-08-05T15:49:36.646 INFO:teuthology.orchestra.run.smithi059.stdout:UBUNTU_CODENAME=focal 2022-08-05T15:49:36.647 INFO:teuthology.lock.ops:Updating smithi059.front.sepia.ceph.com on lock server 2022-08-05T15:49:36.678 DEBUG:teuthology.orchestra.run.smithi157:> uname -m 2022-08-05T15:49:36.696 INFO:teuthology.orchestra.run.smithi157.stdout:x86_64 2022-08-05T15:49:36.697 DEBUG:teuthology.orchestra.run.smithi157:> cat /etc/os-release 2022-08-05T15:49:36.749 INFO:teuthology.orchestra.run.smithi157.stdout:NAME="Ubuntu" 2022-08-05T15:49:36.750 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-08-05T15:49:36.764 INFO:teuthology.orchestra.run.smithi157.stdout:ID=ubuntu 2022-08-05T15:49:36.764 INFO:teuthology.orchestra.run.smithi157.stdout:ID_LIKE=debian 2022-08-05T15:49:36.765 INFO:teuthology.orchestra.run.smithi157.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-08-05T15:49:36.765 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION_ID="20.04" 2022-08-05T15:49:36.776 INFO:teuthology.orchestra.run.smithi157.stdout:HOME_URL="https://www.ubuntu.com/" 2022-08-05T15:49:36.776 INFO:teuthology.orchestra.run.smithi157.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-08-05T15:49:36.777 INFO:teuthology.orchestra.run.smithi157.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-08-05T15:49:36.777 INFO:teuthology.orchestra.run.smithi157.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-08-05T15:49:36.777 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION_CODENAME=focal 2022-08-05T15:49:36.777 INFO:teuthology.orchestra.run.smithi157.stdout:UBUNTU_CODENAME=focal 2022-08-05T15:49:36.778 INFO:teuthology.lock.ops:Updating smithi157.front.sepia.ceph.com on lock server 2022-08-05T15:49:36.800 DEBUG:teuthology.orchestra.run.smithi201:> uname -m 2022-08-05T15:49:36.828 INFO:teuthology.orchestra.run.smithi201.stdout:x86_64 2022-08-05T15:49:36.850 DEBUG:teuthology.orchestra.run.smithi201:> cat /etc/os-release 2022-08-05T15:49:36.909 INFO:teuthology.orchestra.run.smithi201.stdout:NAME="Ubuntu" 2022-08-05T15:49:36.910 INFO:teuthology.orchestra.run.smithi201.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-08-05T15:49:36.910 INFO:teuthology.orchestra.run.smithi201.stdout:ID=ubuntu 2022-08-05T15:49:36.911 INFO:teuthology.orchestra.run.smithi201.stdout:ID_LIKE=debian 2022-08-05T15:49:36.911 INFO:teuthology.orchestra.run.smithi201.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-08-05T15:49:36.912 INFO:teuthology.orchestra.run.smithi201.stdout:VERSION_ID="20.04" 2022-08-05T15:49:36.912 INFO:teuthology.orchestra.run.smithi201.stdout:HOME_URL="https://www.ubuntu.com/" 2022-08-05T15:49:36.918 INFO:teuthology.orchestra.run.smithi201.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-08-05T15:49:36.918 INFO:teuthology.orchestra.run.smithi201.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-08-05T15:49:36.919 INFO:teuthology.orchestra.run.smithi201.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-08-05T15:49:36.919 INFO:teuthology.orchestra.run.smithi201.stdout:VERSION_CODENAME=focal 2022-08-05T15:49:36.920 INFO:teuthology.orchestra.run.smithi201.stdout:UBUNTU_CODENAME=focal 2022-08-05T15:49:36.920 INFO:teuthology.lock.ops:Updating smithi201.front.sepia.ceph.com on lock server 2022-08-05T15:49:36.967 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-08-05T15:49:37.026 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-08-05T15:49:37.075 INFO:teuthology.task.internal:Checking for old test directory... 2022-08-05T15:49:37.076 DEBUG:teuthology.orchestra.run.smithi059:> test '!' -e /home/ubuntu/cephtest 2022-08-05T15:49:37.112 DEBUG:teuthology.orchestra.run.smithi157:> test '!' -e /home/ubuntu/cephtest 2022-08-05T15:49:37.123 DEBUG:teuthology.orchestra.run.smithi201:> test '!' -e /home/ubuntu/cephtest 2022-08-05T15:49:37.144 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-08-05T15:49:37.159 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-08-05T15:49:37.160 DEBUG:teuthology.orchestra.run.smithi059:> test -z $(ls -A /var/lib/ceph) 2022-08-05T15:49:37.173 DEBUG:teuthology.orchestra.run.smithi157:> test -z $(ls -A /var/lib/ceph) 2022-08-05T15:49:37.177 DEBUG:teuthology.orchestra.run.smithi201:> test -z $(ls -A /var/lib/ceph) 2022-08-05T15:49:37.223 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-08-05T15:49:37.334 INFO:teuthology.run_tasks:Running task kernel... 2022-08-05T15:49:37.366 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-08-05T15:49:37.367 INFO:teuthology.task.kernel:normalize config orig: {'hwe': True} 2022-08-05T15:49:37.367 DEBUG:teuthology.task.kernel:normalized overrides {'host.a': {'hwe': True}, 'client.a': {'hwe': True}, 'host.b': {'hwe': True}, 'client.b': {'hwe': True}, 'host.c': {'hwe': True}, 'client.c': {'hwe': True}} 2022-08-05T15:49:37.368 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.b': {'kdb': True, 'sha1': 'distro', 'hwe': True}, 'host.c': {'kdb': True, 'sha1': 'distro', 'hwe': True}}, timeout 300 2022-08-05T15:49:37.368 DEBUG:teuthology.orchestra.run.smithi059:> test -f /run/.containerenv -o -f /.dockerenv 2022-08-05T15:49:37.371 DEBUG:teuthology.orchestra.run.smithi157:> test -f /run/.containerenv -o -f /.dockerenv 2022-08-05T15:49:37.372 DEBUG:teuthology.orchestra.run.smithi201:> test -f /run/.containerenv -o -f /.dockerenv 2022-08-05T15:49:37.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-08-05T15:49:37.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-08-05T15:49:37.381 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2022-08-05T15:49:37.381 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2022-08-05T15:49:37.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-08-05T15:49:37.384 DEBUG:teuthology.orchestra.run.smithi201:> uname -r 2022-08-05T15:49:37.396 INFO:teuthology.orchestra.run.smithi201.stdout:5.4.0-122-generic 2022-08-05T15:49:37.396 INFO:teuthology.task.kernel:Running kernel on smithi201: 5.4.0-122-generic 2022-08-05T15:49:37.397 DEBUG:teuthology.orchestra.run.smithi201:> sudo apt-get clean 2022-08-05T15:49:37.450 INFO:teuthology.orchestra.run.smithi157.stdout:5.4.0-122-generic 2022-08-05T15:49:37.451 INFO:teuthology.task.kernel:Running kernel on smithi157: 5.4.0-122-generic 2022-08-05T15:49:37.460 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-get clean 2022-08-05T15:49:37.465 INFO:teuthology.orchestra.run.smithi059.stdout:5.4.0-122-generic 2022-08-05T15:49:37.465 INFO:teuthology.task.kernel:Running kernel on smithi059: 5.4.0-122-generic 2022-08-05T15:49:37.466 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-get clean 2022-08-05T15:49:37.649 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-get update 2022-08-05T15:49:37.651 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-get update 2022-08-05T15:49:37.673 DEBUG:teuthology.orchestra.run.smithi201:> sudo apt-get update 2022-08-05T15:49:37.832 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-08-05T15:49:37.840 INFO:teuthology.orchestra.run.smithi059.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-08-05T15:49:37.843 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-08-05T15:49:37.851 INFO:teuthology.orchestra.run.smithi157.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-08-05T15:49:37.906 INFO:teuthology.orchestra.run.smithi059.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-08-05T15:49:37.915 INFO:teuthology.orchestra.run.smithi157.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-08-05T15:49:37.942 INFO:teuthology.orchestra.run.smithi059.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-08-05T15:49:37.952 INFO:teuthology.orchestra.run.smithi157.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-08-05T15:49:37.995 INFO:teuthology.orchestra.run.smithi201.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-08-05T15:49:37.996 INFO:teuthology.orchestra.run.smithi201.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-08-05T15:49:38.030 INFO:teuthology.orchestra.run.smithi201.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-08-05T15:49:38.216 INFO:teuthology.orchestra.run.smithi059.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,002 kB] 2022-08-05T15:49:38.227 INFO:teuthology.orchestra.run.smithi157.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [701 kB] 2022-08-05T15:49:38.308 INFO:teuthology.orchestra.run.smithi157.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,002 kB] 2022-08-05T15:49:38.337 INFO:teuthology.orchestra.run.smithi059.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [701 kB] 2022-08-05T15:49:38.352 INFO:teuthology.orchestra.run.smithi059.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [360 kB] 2022-08-05T15:49:38.353 INFO:teuthology.orchestra.run.smithi059.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15.6 kB] 2022-08-05T15:49:38.353 INFO:teuthology.orchestra.run.smithi059.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,187 kB] 2022-08-05T15:49:38.353 INFO:teuthology.orchestra.run.smithi157.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [360 kB] 2022-08-05T15:49:38.354 INFO:teuthology.orchestra.run.smithi157.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15.6 kB] 2022-08-05T15:49:38.357 INFO:teuthology.orchestra.run.smithi157.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,187 kB] 2022-08-05T15:49:38.361 INFO:teuthology.orchestra.run.smithi059.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [26.6 kB] 2022-08-05T15:49:38.362 INFO:teuthology.orchestra.run.smithi059.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [169 kB] 2022-08-05T15:49:38.363 INFO:teuthology.orchestra.run.smithi059.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [584 B] 2022-08-05T15:49:38.363 INFO:teuthology.orchestra.run.smithi059.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [934 kB] 2022-08-05T15:49:38.364 INFO:teuthology.orchestra.run.smithi157.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [26.6 kB] 2022-08-05T15:49:38.364 INFO:teuthology.orchestra.run.smithi157.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [169 kB] 2022-08-05T15:49:38.369 INFO:teuthology.orchestra.run.smithi157.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [584 B] 2022-08-05T15:49:38.370 INFO:teuthology.orchestra.run.smithi157.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [934 kB] 2022-08-05T15:49:38.373 INFO:teuthology.orchestra.run.smithi201.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-08-05T15:49:38.382 INFO:teuthology.orchestra.run.smithi059.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [685 kB] 2022-08-05T15:49:38.382 INFO:teuthology.orchestra.run.smithi059.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [210 kB] 2022-08-05T15:49:38.382 INFO:teuthology.orchestra.run.smithi059.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [21.0 kB] 2022-08-05T15:49:38.383 INFO:teuthology.orchestra.run.smithi059.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [24.4 kB] 2022-08-05T15:49:38.383 INFO:teuthology.orchestra.run.smithi059.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,444 B] 2022-08-05T15:49:38.383 INFO:teuthology.orchestra.run.smithi059.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [588 B] 2022-08-05T15:49:38.384 INFO:teuthology.orchestra.run.smithi059.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.6 kB] 2022-08-05T15:49:38.384 INFO:teuthology.orchestra.run.smithi059.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.0 kB] 2022-08-05T15:49:38.384 INFO:teuthology.orchestra.run.smithi059.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [16.3 kB] 2022-08-05T15:49:38.384 INFO:teuthology.orchestra.run.smithi157.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [685 kB] 2022-08-05T15:49:38.385 INFO:teuthology.orchestra.run.smithi157.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [210 kB] 2022-08-05T15:49:38.385 INFO:teuthology.orchestra.run.smithi157.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [21.0 kB] 2022-08-05T15:49:38.385 INFO:teuthology.orchestra.run.smithi157.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,444 B] 2022-08-05T15:49:38.385 INFO:teuthology.orchestra.run.smithi157.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [24.4 kB] 2022-08-05T15:49:38.386 INFO:teuthology.orchestra.run.smithi157.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [588 B] 2022-08-05T15:49:38.386 INFO:teuthology.orchestra.run.smithi157.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.0 kB] 2022-08-05T15:49:38.386 INFO:teuthology.orchestra.run.smithi157.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.6 kB] 2022-08-05T15:49:38.386 INFO:teuthology.orchestra.run.smithi157.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [16.3 kB] 2022-08-05T15:49:38.403 INFO:teuthology.orchestra.run.smithi157.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2022-08-05T15:49:38.403 INFO:teuthology.orchestra.run.smithi157.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [24.0 kB] 2022-08-05T15:49:38.403 INFO:teuthology.orchestra.run.smithi157.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.4 kB] 2022-08-05T15:49:38.403 INFO:teuthology.orchestra.run.smithi157.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [16.0 kB] 2022-08-05T15:49:38.404 INFO:teuthology.orchestra.run.smithi157.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [860 B] 2022-08-05T15:49:38.404 INFO:teuthology.orchestra.run.smithi059.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2022-08-05T15:49:38.404 INFO:teuthology.orchestra.run.smithi059.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [24.0 kB] 2022-08-05T15:49:38.404 INFO:teuthology.orchestra.run.smithi059.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.4 kB] 2022-08-05T15:49:38.405 INFO:teuthology.orchestra.run.smithi059.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [16.0 kB] 2022-08-05T15:49:38.405 INFO:teuthology.orchestra.run.smithi059.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [860 B] 2022-08-05T15:49:38.453 INFO:teuthology.orchestra.run.smithi201.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [473 kB] 2022-08-05T15:49:38.537 INFO:teuthology.orchestra.run.smithi059.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,638 kB] 2022-08-05T15:49:38.546 INFO:teuthology.orchestra.run.smithi157.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,638 kB] 2022-08-05T15:49:38.593 INFO:teuthology.orchestra.run.smithi201.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [701 kB] 2022-08-05T15:49:38.760 INFO:teuthology.orchestra.run.smithi201.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,638 kB] 2022-08-05T15:49:38.884 INFO:teuthology.orchestra.run.smithi201.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,002 kB] 2022-08-05T15:49:38.929 INFO:teuthology.orchestra.run.smithi201.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [278 kB] 2022-08-05T15:49:38.945 INFO:teuthology.orchestra.run.smithi201.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.8 kB] 2022-08-05T15:49:38.945 INFO:teuthology.orchestra.run.smithi201.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,103 kB] 2022-08-05T15:49:39.014 INFO:teuthology.orchestra.run.smithi059.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [473 kB] 2022-08-05T15:49:39.047 INFO:teuthology.orchestra.run.smithi059.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [278 kB] 2022-08-05T15:49:39.048 INFO:teuthology.orchestra.run.smithi059.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.8 kB] 2022-08-05T15:49:39.051 INFO:teuthology.orchestra.run.smithi059.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,103 kB] 2022-08-05T15:49:39.065 INFO:teuthology.orchestra.run.smithi157.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [473 kB] 2022-08-05T15:49:39.066 INFO:teuthology.orchestra.run.smithi157.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [278 kB] 2022-08-05T15:49:39.066 INFO:teuthology.orchestra.run.smithi157.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.8 kB] 2022-08-05T15:49:39.066 INFO:teuthology.orchestra.run.smithi157.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,103 kB] 2022-08-05T15:49:39.066 INFO:teuthology.orchestra.run.smithi157.stdout:Get:33 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [25.3 kB] 2022-08-05T15:49:39.067 INFO:teuthology.orchestra.run.smithi157.stdout:Get:34 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [157 kB] 2022-08-05T15:49:39.067 INFO:teuthology.orchestra.run.smithi157.stdout:Get:35 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [588 B] 2022-08-05T15:49:39.067 INFO:teuthology.orchestra.run.smithi157.stdout:Get:36 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [712 kB] 2022-08-05T15:49:39.068 INFO:teuthology.orchestra.run.smithi201.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [25.3 kB] 2022-08-05T15:49:39.068 INFO:teuthology.orchestra.run.smithi201.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [157 kB] 2022-08-05T15:49:39.068 INFO:teuthology.orchestra.run.smithi201.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [588 B] 2022-08-05T15:49:39.068 INFO:teuthology.orchestra.run.smithi201.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [557 kB] 2022-08-05T15:49:39.069 INFO:teuthology.orchestra.run.smithi201.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [712 kB] 2022-08-05T15:49:39.090 INFO:teuthology.orchestra.run.smithi201.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [129 kB] 2022-08-05T15:49:39.090 INFO:teuthology.orchestra.run.smithi201.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.7 kB] 2022-08-05T15:49:39.091 INFO:teuthology.orchestra.run.smithi201.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [360 kB] 2022-08-05T15:49:39.091 INFO:teuthology.orchestra.run.smithi201.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15.6 kB] 2022-08-05T15:49:39.091 INFO:teuthology.orchestra.run.smithi201.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [26.6 kB] 2022-08-05T15:49:39.091 INFO:teuthology.orchestra.run.smithi201.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,187 kB] 2022-08-05T15:49:39.093 INFO:teuthology.orchestra.run.smithi157.stdout:Get:37 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [557 kB] 2022-08-05T15:49:39.097 INFO:teuthology.orchestra.run.smithi201.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [22.2 kB] 2022-08-05T15:49:39.097 INFO:teuthology.orchestra.run.smithi201.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,192 B] 2022-08-05T15:49:39.097 INFO:teuthology.orchestra.run.smithi201.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [508 B] 2022-08-05T15:49:39.097 INFO:teuthology.orchestra.run.smithi201.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [169 kB] 2022-08-05T15:49:39.125 INFO:teuthology.orchestra.run.smithi059.stdout:Get:33 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [25.3 kB] 2022-08-05T15:49:39.128 INFO:teuthology.orchestra.run.smithi059.stdout:Get:34 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [157 kB] 2022-08-05T15:49:39.128 INFO:teuthology.orchestra.run.smithi059.stdout:Get:35 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [588 B] 2022-08-05T15:49:39.128 INFO:teuthology.orchestra.run.smithi059.stdout:Get:36 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [557 kB] 2022-08-05T15:49:39.129 INFO:teuthology.orchestra.run.smithi059.stdout:Get:37 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [712 kB] 2022-08-05T15:49:39.129 INFO:teuthology.orchestra.run.smithi059.stdout:Get:38 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [129 kB] 2022-08-05T15:49:39.129 INFO:teuthology.orchestra.run.smithi059.stdout:Get:39 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.7 kB] 2022-08-05T15:49:39.130 INFO:teuthology.orchestra.run.smithi157.stdout:Get:38 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [129 kB] 2022-08-05T15:49:39.131 INFO:teuthology.orchestra.run.smithi157.stdout:Get:39 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.7 kB] 2022-08-05T15:49:39.132 INFO:teuthology.orchestra.run.smithi201.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [584 B] 2022-08-05T15:49:39.132 INFO:teuthology.orchestra.run.smithi201.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [685 kB] 2022-08-05T15:49:39.133 INFO:teuthology.orchestra.run.smithi201.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [934 kB] 2022-08-05T15:49:39.135 INFO:teuthology.orchestra.run.smithi059.stdout:Get:40 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,192 B] 2022-08-05T15:49:39.136 INFO:teuthology.orchestra.run.smithi059.stdout:Get:41 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [22.2 kB] 2022-08-05T15:49:39.136 INFO:teuthology.orchestra.run.smithi059.stdout:Get:42 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [508 B] 2022-08-05T15:49:39.143 INFO:teuthology.orchestra.run.smithi157.stdout:Get:40 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [22.2 kB] 2022-08-05T15:49:39.143 INFO:teuthology.orchestra.run.smithi157.stdout:Get:41 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,192 B] 2022-08-05T15:49:39.144 INFO:teuthology.orchestra.run.smithi157.stdout:Get:42 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [508 B] 2022-08-05T15:49:39.145 INFO:teuthology.orchestra.run.smithi201.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [210 kB] 2022-08-05T15:49:39.146 INFO:teuthology.orchestra.run.smithi201.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [21.0 kB] 2022-08-05T15:49:39.191 INFO:teuthology.orchestra.run.smithi201.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,444 B] 2022-08-05T15:49:39.192 INFO:teuthology.orchestra.run.smithi201.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [24.4 kB] 2022-08-05T15:49:39.192 INFO:teuthology.orchestra.run.smithi201.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [588 B] 2022-08-05T15:49:39.192 INFO:teuthology.orchestra.run.smithi201.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.0 kB] 2022-08-05T15:49:39.192 INFO:teuthology.orchestra.run.smithi201.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.6 kB] 2022-08-05T15:49:39.220 INFO:teuthology.orchestra.run.smithi201.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main Translation-en [16.3 kB] 2022-08-05T15:49:39.286 INFO:teuthology.orchestra.run.smithi201.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2022-08-05T15:49:39.287 INFO:teuthology.orchestra.run.smithi201.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.4 kB] 2022-08-05T15:49:39.287 INFO:teuthology.orchestra.run.smithi201.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [24.0 kB] 2022-08-05T15:49:39.288 INFO:teuthology.orchestra.run.smithi201.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe Translation-en [16.0 kB] 2022-08-05T15:49:39.288 INFO:teuthology.orchestra.run.smithi201.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [860 B] 2022-08-05T15:49:41.467 INFO:teuthology.orchestra.run.smithi157.stdout:Fetched 12.0 MB in 2s (6,443 kB/s) 2022-08-05T15:49:41.700 INFO:teuthology.orchestra.run.smithi059.stdout:Fetched 12.0 MB in 2s (6,141 kB/s) 2022-08-05T15:49:42.046 INFO:teuthology.orchestra.run.smithi201.stdout:Fetched 12.0 MB in 2s (5,001 kB/s) 2022-08-05T15:49:42.671 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2022-08-05T15:49:42.689 DEBUG:teuthology.orchestra.run.smithi157:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-08-05T15:49:42.765 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2022-08-05T15:49:42.941 INFO:teuthology.orchestra.run.smithi157.stdout:Building dependency tree... 2022-08-05T15:49:42.941 INFO:teuthology.orchestra.run.smithi157.stdout:Reading state information... 2022-08-05T15:49:42.960 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2022-08-05T15:49:42.992 DEBUG:teuthology.orchestra.run.smithi059:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-08-05T15:49:43.081 INFO:teuthology.orchestra.run.smithi157.stdout:The following additional packages will be installed: 2022-08-05T15:49:43.081 INFO:teuthology.orchestra.run.smithi157.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-08-05T15:49:43.082 INFO:teuthology.orchestra.run.smithi157.stdout: libusbmuxd6 linux-image-5.15.0-43-generic linux-modules-5.15.0-43-generic 2022-08-05T15:49:43.082 INFO:teuthology.orchestra.run.smithi157.stdout: linux-modules-extra-5.15.0-43-generic thermald upower usbmuxd 2022-08-05T15:49:43.084 INFO:teuthology.orchestra.run.smithi157.stdout:Suggested packages: 2022-08-05T15:49:43.084 INFO:teuthology.orchestra.run.smithi157.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.15-source-5.15.0 2022-08-05T15:49:43.085 INFO:teuthology.orchestra.run.smithi157.stdout: linux-hwe-5.15-tools linux-headers-5.15.0-43-generic 2022-08-05T15:49:43.114 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2022-08-05T15:49:43.133 INFO:teuthology.orchestra.run.smithi157.stdout:The following NEW packages will be installed: 2022-08-05T15:49:43.133 INFO:teuthology.orchestra.run.smithi157.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-08-05T15:49:43.133 INFO:teuthology.orchestra.run.smithi157.stdout: libusbmuxd6 linux-image-5.15.0-43-generic linux-image-generic-hwe-20.04 2022-08-05T15:49:43.134 INFO:teuthology.orchestra.run.smithi157.stdout: linux-modules-5.15.0-43-generic linux-modules-extra-5.15.0-43-generic 2022-08-05T15:49:43.134 INFO:teuthology.orchestra.run.smithi157.stdout: thermald upower usbmuxd 2022-08-05T15:49:43.185 INFO:teuthology.orchestra.run.smithi157.stdout:0 upgraded, 13 newly installed, 0 to remove and 19 not upgraded. 2022-08-05T15:49:43.186 INFO:teuthology.orchestra.run.smithi157.stdout:Need to get 87.9 MB of archives. 2022-08-05T15:49:43.186 INFO:teuthology.orchestra.run.smithi157.stdout:After this operation, 463 MB of additional disk space will be used. 2022-08-05T15:49:43.186 INFO:teuthology.orchestra.run.smithi157.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-08-05T15:49:43.247 INFO:teuthology.orchestra.run.smithi157.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-08-05T15:49:43.255 INFO:teuthology.orchestra.run.smithi157.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-08-05T15:49:43.255 INFO:teuthology.orchestra.run.smithi157.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-08-05T15:49:43.269 INFO:teuthology.orchestra.run.smithi157.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-08-05T15:49:43.277 INFO:teuthology.orchestra.run.smithi201.stdout:Reading package lists... 2022-08-05T15:49:43.281 INFO:teuthology.orchestra.run.smithi157.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.15.0-43-generic amd64 5.15.0-43.46~20.04.1 [20.0 MB] 2022-08-05T15:49:43.299 INFO:teuthology.orchestra.run.smithi059.stdout:Building dependency tree... 2022-08-05T15:49:43.300 INFO:teuthology.orchestra.run.smithi059.stdout:Reading state information... 2022-08-05T15:49:43.302 DEBUG:teuthology.orchestra.run.smithi201:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-08-05T15:49:43.379 INFO:teuthology.orchestra.run.smithi201.stdout:Reading package lists... 2022-08-05T15:49:43.456 INFO:teuthology.orchestra.run.smithi059.stdout:The following additional packages will be installed: 2022-08-05T15:49:43.457 INFO:teuthology.orchestra.run.smithi059.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-08-05T15:49:43.457 INFO:teuthology.orchestra.run.smithi059.stdout: libusbmuxd6 linux-image-5.15.0-43-generic linux-modules-5.15.0-43-generic 2022-08-05T15:49:43.458 INFO:teuthology.orchestra.run.smithi059.stdout: linux-modules-extra-5.15.0-43-generic thermald upower usbmuxd 2022-08-05T15:49:43.459 INFO:teuthology.orchestra.run.smithi059.stdout:Suggested packages: 2022-08-05T15:49:43.459 INFO:teuthology.orchestra.run.smithi059.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.15-source-5.15.0 2022-08-05T15:49:43.460 INFO:teuthology.orchestra.run.smithi059.stdout: linux-hwe-5.15-tools linux-headers-5.15.0-43-generic 2022-08-05T15:49:43.510 INFO:teuthology.orchestra.run.smithi059.stdout:The following NEW packages will be installed: 2022-08-05T15:49:43.511 INFO:teuthology.orchestra.run.smithi059.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-08-05T15:49:43.511 INFO:teuthology.orchestra.run.smithi059.stdout: libusbmuxd6 linux-image-5.15.0-43-generic linux-image-generic-hwe-20.04 2022-08-05T15:49:43.512 INFO:teuthology.orchestra.run.smithi059.stdout: linux-modules-5.15.0-43-generic linux-modules-extra-5.15.0-43-generic 2022-08-05T15:49:43.512 INFO:teuthology.orchestra.run.smithi059.stdout: thermald upower usbmuxd 2022-08-05T15:49:43.555 INFO:teuthology.orchestra.run.smithi201.stdout:Building dependency tree... 2022-08-05T15:49:43.555 INFO:teuthology.orchestra.run.smithi201.stdout:Reading state information... 2022-08-05T15:49:43.573 INFO:teuthology.orchestra.run.smithi157.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.15.0-43-generic amd64 5.15.0-43.46~20.04.1 [10.8 MB] 2022-08-05T15:49:43.670 INFO:teuthology.orchestra.run.smithi157.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.15.0-43-generic amd64 5.15.0-43.46~20.04.1 [56.4 MB] 2022-08-05T15:49:43.691 INFO:teuthology.orchestra.run.smithi059.stdout:0 upgraded, 13 newly installed, 0 to remove and 19 not upgraded. 2022-08-05T15:49:43.692 INFO:teuthology.orchestra.run.smithi059.stdout:Need to get 87.9 MB of archives. 2022-08-05T15:49:43.692 INFO:teuthology.orchestra.run.smithi059.stdout:After this operation, 463 MB of additional disk space will be used. 2022-08-05T15:49:43.692 INFO:teuthology.orchestra.run.smithi059.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-08-05T15:49:43.724 INFO:teuthology.orchestra.run.smithi201.stdout:The following additional packages will be installed: 2022-08-05T15:49:43.725 INFO:teuthology.orchestra.run.smithi201.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-08-05T15:49:43.725 INFO:teuthology.orchestra.run.smithi201.stdout: libusbmuxd6 linux-image-5.15.0-43-generic linux-modules-5.15.0-43-generic 2022-08-05T15:49:43.725 INFO:teuthology.orchestra.run.smithi201.stdout: linux-modules-extra-5.15.0-43-generic thermald upower usbmuxd 2022-08-05T15:49:43.726 INFO:teuthology.orchestra.run.smithi201.stdout:Suggested packages: 2022-08-05T15:49:43.726 INFO:teuthology.orchestra.run.smithi201.stdout: libusbmuxd-tools fdutils linux-doc | linux-hwe-5.15-source-5.15.0 2022-08-05T15:49:43.727 INFO:teuthology.orchestra.run.smithi201.stdout: linux-hwe-5.15-tools linux-headers-5.15.0-43-generic 2022-08-05T15:49:43.778 INFO:teuthology.orchestra.run.smithi201.stdout:The following NEW packages will be installed: 2022-08-05T15:49:43.779 INFO:teuthology.orchestra.run.smithi201.stdout: libdbus-glib-1-2 libevdev2 libimobiledevice6 libplist3 libupower-glib3 2022-08-05T15:49:43.780 INFO:teuthology.orchestra.run.smithi201.stdout: libusbmuxd6 linux-image-5.15.0-43-generic linux-image-generic-hwe-20.04 2022-08-05T15:49:43.780 INFO:teuthology.orchestra.run.smithi201.stdout: linux-modules-5.15.0-43-generic linux-modules-extra-5.15.0-43-generic 2022-08-05T15:49:43.781 INFO:teuthology.orchestra.run.smithi201.stdout: thermald upower usbmuxd 2022-08-05T15:49:43.820 INFO:teuthology.orchestra.run.smithi201.stdout:0 upgraded, 13 newly installed, 0 to remove and 19 not upgraded. 2022-08-05T15:49:43.820 INFO:teuthology.orchestra.run.smithi201.stdout:Need to get 87.9 MB of archives. 2022-08-05T15:49:43.820 INFO:teuthology.orchestra.run.smithi201.stdout:After this operation, 463 MB of additional disk space will be used. 2022-08-05T15:49:43.821 INFO:teuthology.orchestra.run.smithi201.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB] 2022-08-05T15:49:43.883 INFO:teuthology.orchestra.run.smithi201.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-08-05T15:49:43.886 INFO:teuthology.orchestra.run.smithi201.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-08-05T15:49:43.892 INFO:teuthology.orchestra.run.smithi201.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-08-05T15:49:43.911 INFO:teuthology.orchestra.run.smithi201.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-08-05T15:49:43.912 INFO:teuthology.orchestra.run.smithi201.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.15.0-43-generic amd64 5.15.0-43.46~20.04.1 [20.0 MB] 2022-08-05T15:49:43.967 INFO:teuthology.orchestra.run.smithi059.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libplist3 amd64 2.1.0-4build2 [31.6 kB] 2022-08-05T15:49:43.997 INFO:teuthology.orchestra.run.smithi059.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libusbmuxd6 amd64 2.0.1-2 [19.1 kB] 2022-08-05T15:49:44.015 INFO:teuthology.orchestra.run.smithi059.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libimobiledevice6 amd64 1.2.1~git20191129.9f79242-1build1 [65.2 kB] 2022-08-05T15:49:44.083 INFO:teuthology.orchestra.run.smithi059.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libupower-glib3 amd64 0.99.11-1build2 [43.2 kB] 2022-08-05T15:49:44.120 INFO:teuthology.orchestra.run.smithi059.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.15.0-43-generic amd64 5.15.0-43.46~20.04.1 [20.0 MB] 2022-08-05T15:49:44.201 INFO:teuthology.orchestra.run.smithi201.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.15.0-43-generic amd64 5.15.0-43.46~20.04.1 [10.8 MB] 2022-08-05T15:49:44.303 INFO:teuthology.orchestra.run.smithi201.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.15.0-43-generic amd64 5.15.0-43.46~20.04.1 [56.4 MB] 2022-08-05T15:49:44.359 INFO:teuthology.orchestra.run.smithi157.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.15.0.43.46~20.04.14 [2,676 B] 2022-08-05T15:49:44.360 INFO:teuthology.orchestra.run.smithi157.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.2 [31.6 kB] 2022-08-05T15:49:44.361 INFO:teuthology.orchestra.run.smithi157.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-08-05T15:49:44.375 INFO:teuthology.orchestra.run.smithi157.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-08-05T15:49:44.419 INFO:teuthology.orchestra.run.smithi157.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-08-05T15:49:44.817 INFO:teuthology.orchestra.run.smithi157.stdout:Fetched 87.9 MB in 1s (68.7 MB/s) 2022-08-05T15:49:44.973 INFO:teuthology.orchestra.run.smithi059.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.15.0-43-generic amd64 5.15.0-43.46~20.04.1 [10.8 MB] 2022-08-05T15:49:44.985 INFO:teuthology.orchestra.run.smithi201.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.15.0.43.46~20.04.14 [2,676 B] 2022-08-05T15:49:44.985 INFO:teuthology.orchestra.run.smithi201.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.2 [31.6 kB] 2022-08-05T15:49:45.050 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-08-05T15:49:45.064 INFO:teuthology.orchestra.run.smithi201.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-08-05T15:49:45.064 INFO:teuthology.orchestra.run.smithi201.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-08-05T15:49:45.065 INFO:teuthology.orchestra.run.smithi201.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-08-05T15:49:45.297 INFO:teuthology.orchestra.run.smithi059.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.15.0-43-generic amd64 5.15.0-43.46~20.04.1 [56.4 MB] 2022-08-05T15:49:45.491 INFO:teuthology.orchestra.run.smithi201.stdout:Fetched 87.9 MB in 1s (68.2 MB/s) 2022-08-05T15:49:45.663 INFO:teuthology.orchestra.run.smithi201.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-08-05T15:49:46.399 INFO:teuthology.orchestra.run.smithi157.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102338 files and directories currently installed.) 2022-08-05T15:49:46.400 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-08-05T15:49:46.524 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-08-05T15:49:46.894 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libplist3:amd64. 2022-08-05T15:49:46.902 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-08-05T15:49:46.937 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-08-05T15:49:46.952 INFO:teuthology.orchestra.run.smithi201.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102338 files and directories currently installed.) 2022-08-05T15:49:46.955 INFO:teuthology.orchestra.run.smithi201.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-08-05T15:49:47.084 INFO:teuthology.orchestra.run.smithi059.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic-hwe-20.04 amd64 5.15.0.43.46~20.04.14 [2,676 B] 2022-08-05T15:49:47.085 INFO:teuthology.orchestra.run.smithi059.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libevdev2 amd64 1.9.0+dfsg-1ubuntu0.2 [31.6 kB] 2022-08-05T15:49:47.085 INFO:teuthology.orchestra.run.smithi059.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 thermald amd64 1.9.1-1ubuntu0.6 [233 kB] 2022-08-05T15:49:47.088 INFO:teuthology.orchestra.run.smithi201.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-08-05T15:49:47.088 INFO:teuthology.orchestra.run.smithi059.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 upower amd64 0.99.11-1build2 [104 kB] 2022-08-05T15:49:47.090 INFO:teuthology.orchestra.run.smithi059.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 usbmuxd amd64 1.1.1~git20191130.9af2b12-1 [38.4 kB] 2022-08-05T15:49:47.282 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-08-05T15:49:47.294 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-08-05T15:49:47.348 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-08-05T15:49:47.428 INFO:teuthology.orchestra.run.smithi201.stdout:Selecting previously unselected package libplist3:amd64. 2022-08-05T15:49:47.442 INFO:teuthology.orchestra.run.smithi201.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-08-05T15:49:47.509 INFO:teuthology.orchestra.run.smithi201.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-08-05T15:49:47.522 INFO:teuthology.orchestra.run.smithi059.stdout:Fetched 87.9 MB in 4s (24.6 MB/s) 2022-08-05T15:49:47.655 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-08-05T15:49:47.670 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-08-05T15:49:47.709 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-08-05T15:49:47.765 INFO:teuthology.orchestra.run.smithi201.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-08-05T15:49:47.791 INFO:teuthology.orchestra.run.smithi201.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-08-05T15:49:47.814 INFO:teuthology.orchestra.run.smithi201.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-08-05T15:49:47.847 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libdbus-glib-1-2:amd64. 2022-08-05T15:49:47.978 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-08-05T15:49:47.995 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-08-05T15:49:48.039 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-08-05T15:49:48.143 INFO:teuthology.orchestra.run.smithi201.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-08-05T15:49:48.172 INFO:teuthology.orchestra.run.smithi201.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-08-05T15:49:48.204 INFO:teuthology.orchestra.run.smithi201.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-08-05T15:49:48.362 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package linux-modules-5.15.0-43-generic. 2022-08-05T15:49:48.393 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../05-linux-modules-5.15.0-43-generic_5.15.0-43.46~20.04.1_amd64.deb ... 2022-08-05T15:49:48.428 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-modules-5.15.0-43-generic (5.15.0-43.46~20.04.1) ... 2022-08-05T15:49:48.574 INFO:teuthology.orchestra.run.smithi201.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-08-05T15:49:48.617 INFO:teuthology.orchestra.run.smithi201.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-08-05T15:49:48.632 INFO:teuthology.orchestra.run.smithi201.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-08-05T15:49:48.899 INFO:teuthology.orchestra.run.smithi201.stdout:Selecting previously unselected package linux-modules-5.15.0-43-generic. 2022-08-05T15:49:48.915 INFO:teuthology.orchestra.run.smithi201.stdout:Preparing to unpack .../05-linux-modules-5.15.0-43-generic_5.15.0-43.46~20.04.1_amd64.deb ... 2022-08-05T15:49:48.975 INFO:teuthology.orchestra.run.smithi201.stdout:Unpacking linux-modules-5.15.0-43-generic (5.15.0-43.46~20.04.1) ... 2022-08-05T15:49:49.218 INFO:teuthology.orchestra.run.smithi059.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102338 files and directories currently installed.) 2022-08-05T15:49:49.239 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../00-libdbus-glib-1-2_0.110-5fakssync1_amd64.deb ... 2022-08-05T15:49:49.386 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-08-05T15:49:49.786 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libplist3:amd64. 2022-08-05T15:49:49.802 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../01-libplist3_2.1.0-4build2_amd64.deb ... 2022-08-05T15:49:49.842 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libplist3:amd64 (2.1.0-4build2) ... 2022-08-05T15:49:50.146 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libusbmuxd6:amd64. 2022-08-05T15:49:50.191 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../02-libusbmuxd6_2.0.1-2_amd64.deb ... 2022-08-05T15:49:50.191 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libusbmuxd6:amd64 (2.0.1-2) ... 2022-08-05T15:49:50.523 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libimobiledevice6:amd64. 2022-08-05T15:49:50.544 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../03-libimobiledevice6_1.2.1~git20191129.9f79242-1build1_amd64.deb ... 2022-08-05T15:49:50.590 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-08-05T15:49:50.918 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libupower-glib3:amd64. 2022-08-05T15:49:50.932 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../04-libupower-glib3_0.99.11-1build2_amd64.deb ... 2022-08-05T15:49:50.977 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package linux-image-5.15.0-43-generic. 2022-08-05T15:49:50.979 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../06-linux-image-5.15.0-43-generic_5.15.0-43.46~20.04.1_amd64.deb ... 2022-08-05T15:49:50.992 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libupower-glib3:amd64 (0.99.11-1build2) ... 2022-08-05T15:49:51.114 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-image-5.15.0-43-generic (5.15.0-43.46~20.04.1) ... 2022-08-05T15:49:51.163 INFO:teuthology.orchestra.run.smithi201.stdout:Selecting previously unselected package linux-image-5.15.0-43-generic. 2022-08-05T15:49:51.164 INFO:teuthology.orchestra.run.smithi201.stdout:Preparing to unpack .../06-linux-image-5.15.0-43-generic_5.15.0-43.46~20.04.1_amd64.deb ... 2022-08-05T15:49:51.265 INFO:teuthology.orchestra.run.smithi201.stdout:Unpacking linux-image-5.15.0-43-generic (5.15.0-43.46~20.04.1) ... 2022-08-05T15:49:51.279 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package linux-modules-5.15.0-43-generic. 2022-08-05T15:49:51.296 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../05-linux-modules-5.15.0-43-generic_5.15.0-43.46~20.04.1_amd64.deb ... 2022-08-05T15:49:51.355 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-modules-5.15.0-43-generic (5.15.0-43.46~20.04.1) ... 2022-08-05T15:49:51.560 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-43-generic. 2022-08-05T15:49:51.577 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../07-linux-modules-extra-5.15.0-43-generic_5.15.0-43.46~20.04.1_amd64.deb ... 2022-08-05T15:49:51.613 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-modules-extra-5.15.0-43-generic (5.15.0-43.46~20.04.1) ... 2022-08-05T15:49:51.709 INFO:teuthology.orchestra.run.smithi201.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-43-generic. 2022-08-05T15:49:51.730 INFO:teuthology.orchestra.run.smithi201.stdout:Preparing to unpack .../07-linux-modules-extra-5.15.0-43-generic_5.15.0-43.46~20.04.1_amd64.deb ... 2022-08-05T15:49:51.763 INFO:teuthology.orchestra.run.smithi201.stdout:Unpacking linux-modules-extra-5.15.0-43-generic (5.15.0-43.46~20.04.1) ... 2022-08-05T15:49:53.688 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package linux-image-5.15.0-43-generic. 2022-08-05T15:49:53.706 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../06-linux-image-5.15.0-43-generic_5.15.0-43.46~20.04.1_amd64.deb ... 2022-08-05T15:49:53.798 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-image-5.15.0-43-generic (5.15.0-43.46~20.04.1) ... 2022-08-05T15:49:54.327 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-43-generic. 2022-08-05T15:49:54.342 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../07-linux-modules-extra-5.15.0-43-generic_5.15.0-43.46~20.04.1_amd64.deb ... 2022-08-05T15:49:54.382 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-modules-extra-5.15.0-43-generic (5.15.0-43.46~20.04.1) ... 2022-08-05T15:49:57.971 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-08-05T15:49:57.986 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.15.0.43.46~20.04.14_amd64.deb ... 2022-08-05T15:49:58.037 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-image-generic-hwe-20.04 (5.15.0.43.46~20.04.14) ... 2022-08-05T15:49:58.128 INFO:teuthology.orchestra.run.smithi201.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-08-05T15:49:58.143 INFO:teuthology.orchestra.run.smithi201.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.15.0.43.46~20.04.14_amd64.deb ... 2022-08-05T15:49:58.173 INFO:teuthology.orchestra.run.smithi201.stdout:Unpacking linux-image-generic-hwe-20.04 (5.15.0.43.46~20.04.14) ... 2022-08-05T15:49:58.323 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libevdev2:amd64. 2022-08-05T15:49:58.343 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.2_amd64.deb ... 2022-08-05T15:49:58.388 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2022-08-05T15:49:58.468 INFO:teuthology.orchestra.run.smithi201.stdout:Selecting previously unselected package libevdev2:amd64. 2022-08-05T15:49:58.483 INFO:teuthology.orchestra.run.smithi201.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.2_amd64.deb ... 2022-08-05T15:49:58.530 INFO:teuthology.orchestra.run.smithi201.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2022-08-05T15:49:58.750 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package thermald. 2022-08-05T15:49:58.763 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-08-05T15:49:58.808 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-08-05T15:49:58.874 INFO:teuthology.orchestra.run.smithi201.stdout:Selecting previously unselected package thermald. 2022-08-05T15:49:58.885 INFO:teuthology.orchestra.run.smithi201.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-08-05T15:49:58.915 INFO:teuthology.orchestra.run.smithi201.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-08-05T15:49:59.357 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package upower. 2022-08-05T15:49:59.372 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-08-05T15:49:59.442 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking upower (0.99.11-1build2) ... 2022-08-05T15:49:59.454 INFO:teuthology.orchestra.run.smithi201.stdout:Selecting previously unselected package upower. 2022-08-05T15:49:59.466 INFO:teuthology.orchestra.run.smithi201.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-08-05T15:49:59.513 INFO:teuthology.orchestra.run.smithi201.stdout:Unpacking upower (0.99.11-1build2) ... 2022-08-05T15:49:59.874 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package usbmuxd. 2022-08-05T15:49:59.882 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-08-05T15:49:59.923 INFO:teuthology.orchestra.run.smithi201.stdout:Selecting previously unselected package usbmuxd. 2022-08-05T15:49:59.928 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-08-05T15:49:59.946 INFO:teuthology.orchestra.run.smithi201.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-08-05T15:49:59.982 INFO:teuthology.orchestra.run.smithi201.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-08-05T15:50:00.262 INFO:teuthology.orchestra.run.smithi201.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-08-05T15:50:00.307 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-08-05T15:50:00.377 INFO:teuthology.orchestra.run.smithi201.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-08-05T15:50:00.435 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-08-05T15:50:00.492 INFO:teuthology.orchestra.run.smithi201.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-08-05T15:50:00.559 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-08-05T15:50:00.618 INFO:teuthology.orchestra.run.smithi201.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-08-05T15:50:00.682 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-08-05T15:50:00.767 INFO:teuthology.orchestra.run.smithi201.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-08-05T15:50:00.787 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package linux-image-generic-hwe-20.04. 2022-08-05T15:50:00.814 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../08-linux-image-generic-hwe-20.04_5.15.0.43.46~20.04.14_amd64.deb ... 2022-08-05T15:50:00.815 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-08-05T15:50:00.846 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking linux-image-generic-hwe-20.04 (5.15.0.43.46~20.04.14) ... 2022-08-05T15:50:00.870 INFO:teuthology.orchestra.run.smithi201.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2022-08-05T15:50:00.953 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2022-08-05T15:50:01.023 INFO:teuthology.orchestra.run.smithi201.stdout:Setting up upower (0.99.11-1build2) ... 2022-08-05T15:50:01.092 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up upower (0.99.11-1build2) ... 2022-08-05T15:50:01.190 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package libevdev2:amd64. 2022-08-05T15:50:01.203 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../09-libevdev2_1.9.0+dfsg-1ubuntu0.2_amd64.deb ... 2022-08-05T15:50:01.244 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2022-08-05T15:50:01.427 INFO:teuthology.orchestra.run.smithi201.stdout:upower.service is a disabled or a static unit, not starting it. 2022-08-05T15:50:01.479 INFO:teuthology.orchestra.run.smithi157.stdout:upower.service is a disabled or a static unit, not starting it. 2022-08-05T15:50:01.479 INFO:teuthology.orchestra.run.smithi201.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-08-05T15:50:01.525 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-08-05T15:50:01.576 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package thermald. 2022-08-05T15:50:01.594 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../10-thermald_1.9.1-1ubuntu0.6_amd64.deb ... 2022-08-05T15:50:01.635 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking thermald (1.9.1-1ubuntu0.6) ... 2022-08-05T15:50:01.640 INFO:teuthology.orchestra.run.smithi201.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-08-05T15:50:01.653 INFO:teuthology.orchestra.run.smithi201.stdout:Adding system user `usbmux' (UID 114) ... 2022-08-05T15:50:01.654 INFO:teuthology.orchestra.run.smithi201.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-08-05T15:50:01.700 INFO:teuthology.orchestra.run.smithi157.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-08-05T15:50:01.700 INFO:teuthology.orchestra.run.smithi157.stdout:Adding system user `usbmux' (UID 114) ... 2022-08-05T15:50:01.700 INFO:teuthology.orchestra.run.smithi157.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-08-05T15:50:02.178 INFO:teuthology.orchestra.run.smithi201.stdout:Not creating home directory `/var/lib/usbmux'. 2022-08-05T15:50:02.269 INFO:teuthology.orchestra.run.smithi157.stdout:Not creating home directory `/var/lib/usbmux'. 2022-08-05T15:50:02.394 INFO:teuthology.orchestra.run.smithi201.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-08-05T15:50:02.438 INFO:teuthology.orchestra.run.smithi201.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-08-05T15:50:02.475 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package upower. 2022-08-05T15:50:02.476 INFO:teuthology.orchestra.run.smithi157.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-08-05T15:50:02.488 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../11-upower_0.99.11-1build2_amd64.deb ... 2022-08-05T15:50:02.516 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-08-05T15:50:02.588 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking upower (0.99.11-1build2) ... 2022-08-05T15:50:02.740 INFO:teuthology.orchestra.run.smithi201.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-08-05T15:50:02.741 INFO:teuthology.orchestra.run.smithi201.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-08-05T15:50:02.856 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-08-05T15:50:02.857 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-08-05T15:50:02.949 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package usbmuxd. 2022-08-05T15:50:02.967 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../12-usbmuxd_1.1.1~git20191130.9af2b12-1_amd64.deb ... 2022-08-05T15:50:03.003 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-08-05T15:50:03.181 INFO:teuthology.orchestra.run.smithi201.stdout:Setting up linux-image-5.15.0-43-generic (5.15.0-43.46~20.04.1) ... 2022-08-05T15:50:03.255 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libplist3:amd64 (2.1.0-4build2) ... 2022-08-05T15:50:03.333 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-image-5.15.0-43-generic (5.15.0-43.46~20.04.1) ... 2022-08-05T15:50:03.395 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libdbus-glib-1-2:amd64 (0.110-5fakssync1) ... 2022-08-05T15:50:03.506 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libusbmuxd6:amd64 (2.0.1-2) ... 2022-08-05T15:50:03.663 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libupower-glib3:amd64 (0.99.11-1build2) ... 2022-08-05T15:50:03.789 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libimobiledevice6:amd64 (1.2.1~git20191129.9f79242-1build1) ... 2022-08-05T15:50:03.915 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up libevdev2:amd64 (1.9.0+dfsg-1ubuntu0.2) ... 2022-08-05T15:50:04.080 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up upower (0.99.11-1build2) ... 2022-08-05T15:50:04.506 INFO:teuthology.orchestra.run.smithi059.stdout:upower.service is a disabled or a static unit, not starting it. 2022-08-05T15:50:04.507 INFO:teuthology.orchestra.run.smithi201.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-43-generic 2022-08-05T15:50:04.507 INFO:teuthology.orchestra.run.smithi201.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-43-generic 2022-08-05T15:50:04.576 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up usbmuxd (1.1.1~git20191130.9af2b12-1) ... 2022-08-05T15:50:04.578 INFO:teuthology.orchestra.run.smithi157.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-43-generic 2022-08-05T15:50:04.578 INFO:teuthology.orchestra.run.smithi157.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-43-generic 2022-08-05T15:50:04.659 INFO:teuthology.orchestra.run.smithi201.stdout:Setting up linux-modules-5.15.0-43-generic (5.15.0-43.46~20.04.1) ... 2022-08-05T15:50:04.750 INFO:teuthology.orchestra.run.smithi059.stdout:Warning: The home dir /var/lib/usbmux you specified can't be accessed: No such file or directory 2022-08-05T15:50:04.754 INFO:teuthology.orchestra.run.smithi059.stdout:Adding system user `usbmux' (UID 114) ... 2022-08-05T15:50:04.755 INFO:teuthology.orchestra.run.smithi059.stdout:Adding new user `usbmux' (UID 114) with group `plugdev' ... 2022-08-05T15:50:04.770 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-modules-5.15.0-43-generic (5.15.0-43.46~20.04.1) ... 2022-08-05T15:50:05.229 INFO:teuthology.orchestra.run.smithi059.stdout:Not creating home directory `/var/lib/usbmux'. 2022-08-05T15:50:05.432 INFO:teuthology.orchestra.run.smithi059.stdout:usbmuxd.service is a disabled or a static unit, not starting it. 2022-08-05T15:50:05.492 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up thermald (1.9.1-1ubuntu0.6) ... 2022-08-05T15:50:05.792 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.thermald.service -> /lib/systemd/system/thermald.service. 2022-08-05T15:50:05.793 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/thermald.service -> /lib/systemd/system/thermald.service. 2022-08-05T15:50:06.035 INFO:teuthology.orchestra.run.smithi201.stdout:Setting up linux-modules-extra-5.15.0-43-generic (5.15.0-43.46~20.04.1) ... 2022-08-05T15:50:06.132 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-modules-extra-5.15.0-43-generic (5.15.0-43.46~20.04.1) ... 2022-08-05T15:50:06.226 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-image-5.15.0-43-generic (5.15.0-43.46~20.04.1) ... 2022-08-05T15:50:07.390 INFO:teuthology.orchestra.run.smithi201.stdout:Setting up linux-image-generic-hwe-20.04 (5.15.0.43.46~20.04.14) ... 2022-08-05T15:50:07.526 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-image-generic-hwe-20.04 (5.15.0.43.46~20.04.14) ... 2022-08-05T15:50:07.528 INFO:teuthology.orchestra.run.smithi201.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-08-05T15:50:07.563 INFO:teuthology.orchestra.run.smithi059.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-43-generic 2022-08-05T15:50:07.564 INFO:teuthology.orchestra.run.smithi059.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-43-generic 2022-08-05T15:50:07.610 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-08-05T15:50:07.723 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-modules-5.15.0-43-generic (5.15.0-43.46~20.04.1) ... 2022-08-05T15:50:09.144 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-modules-extra-5.15.0-43-generic (5.15.0-43.46~20.04.1) ... 2022-08-05T15:50:10.560 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up linux-image-generic-hwe-20.04 (5.15.0.43.46~20.04.14) ... 2022-08-05T15:50:10.710 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-08-05T15:50:11.267 INFO:teuthology.orchestra.run.smithi201.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-08-05T15:50:11.936 INFO:teuthology.orchestra.run.smithi201.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.2) ... 2022-08-05T15:50:12.047 INFO:teuthology.orchestra.run.smithi201.stdout:Processing triggers for linux-image-5.15.0-43-generic (5.15.0-43.46~20.04.1) ... 2022-08-05T15:50:12.136 INFO:teuthology.orchestra.run.smithi201.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-08-05T15:50:12.137 INFO:teuthology.orchestra.run.smithi201.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-43-generic 2022-08-05T15:50:12.358 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-08-05T15:50:13.021 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.2) ... 2022-08-05T15:50:13.104 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for linux-image-5.15.0-43-generic (5.15.0-43.46~20.04.1) ... 2022-08-05T15:50:13.175 INFO:teuthology.orchestra.run.smithi157.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-08-05T15:50:13.175 INFO:teuthology.orchestra.run.smithi157.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-43-generic 2022-08-05T15:50:15.810 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-08-05T15:50:16.495 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.2) ... 2022-08-05T15:50:16.593 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for linux-image-5.15.0-43-generic (5.15.0-43.46~20.04.1) ... 2022-08-05T15:50:16.669 INFO:teuthology.orchestra.run.smithi059.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-08-05T15:50:16.671 INFO:teuthology.orchestra.run.smithi059.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-43-generic 2022-08-05T15:50:36.624 INFO:teuthology.orchestra.run.smithi201.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-08-05T15:50:36.625 INFO:teuthology.orchestra.run.smithi201.stdout:Sourcing file `/etc/default/grub' 2022-08-05T15:50:36.633 INFO:teuthology.orchestra.run.smithi201.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-08-05T15:50:36.650 INFO:teuthology.orchestra.run.smithi201.stdout:Generating grub configuration file ... 2022-08-05T15:50:36.979 INFO:teuthology.orchestra.run.smithi201.stdout:Found linux image: /boot/vmlinuz-5.15.0-43-generic 2022-08-05T15:50:37.006 INFO:teuthology.orchestra.run.smithi201.stdout:Found initrd image: /boot/initrd.img-5.15.0-43-generic 2022-08-05T15:50:37.392 INFO:teuthology.orchestra.run.smithi201.stdout:Found linux image: /boot/vmlinuz-5.4.0-122-generic 2022-08-05T15:50:37.397 INFO:teuthology.orchestra.run.smithi201.stdout:Found initrd image: /boot/initrd.img-5.4.0-122-generic 2022-08-05T15:50:38.277 INFO:teuthology.orchestra.run.smithi157.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-08-05T15:50:38.278 INFO:teuthology.orchestra.run.smithi157.stdout:Sourcing file `/etc/default/grub' 2022-08-05T15:50:38.309 INFO:teuthology.orchestra.run.smithi157.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-08-05T15:50:38.342 INFO:teuthology.orchestra.run.smithi157.stdout:Generating grub configuration file ... 2022-08-05T15:50:38.659 INFO:teuthology.orchestra.run.smithi157.stdout:Found linux image: /boot/vmlinuz-5.15.0-43-generic 2022-08-05T15:50:38.706 INFO:teuthology.orchestra.run.smithi157.stdout:Found initrd image: /boot/initrd.img-5.15.0-43-generic 2022-08-05T15:50:39.101 INFO:teuthology.orchestra.run.smithi157.stdout:Found linux image: /boot/vmlinuz-5.4.0-122-generic 2022-08-05T15:50:39.107 INFO:teuthology.orchestra.run.smithi157.stdout:Found initrd image: /boot/initrd.img-5.4.0-122-generic 2022-08-05T15:50:39.375 INFO:teuthology.orchestra.run.smithi201.stdout:done 2022-08-05T15:50:39.731 DEBUG:teuthology.orchestra.run.smithi201:> dpkg -s linux-image-generic-hwe-20.04 2022-08-05T15:50:39.772 INFO:teuthology.orchestra.run.smithi201.stdout:Package: linux-image-generic-hwe-20.04 2022-08-05T15:50:39.773 INFO:teuthology.orchestra.run.smithi201.stdout:Status: install ok installed 2022-08-05T15:50:39.778 INFO:teuthology.orchestra.run.smithi201.stdout:Priority: optional 2022-08-05T15:50:39.779 INFO:teuthology.orchestra.run.smithi201.stdout:Section: kernel 2022-08-05T15:50:39.779 INFO:teuthology.orchestra.run.smithi201.stdout:Installed-Size: 19 2022-08-05T15:50:39.779 INFO:teuthology.orchestra.run.smithi201.stdout:Maintainer: Ubuntu Kernel Team 2022-08-05T15:50:39.780 INFO:teuthology.orchestra.run.smithi201.stdout:Architecture: amd64 2022-08-05T15:50:39.780 INFO:teuthology.orchestra.run.smithi201.stdout:Source: linux-meta-hwe-5.15 2022-08-05T15:50:39.781 INFO:teuthology.orchestra.run.smithi201.stdout:Version: 5.15.0.43.46~20.04.14 2022-08-05T15:50:39.781 INFO:teuthology.orchestra.run.smithi201.stdout:Provides: virtualbox-guest-modules (= 5.15.0-43), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.4-0ubuntu0.1) 2022-08-05T15:50:39.781 INFO:teuthology.orchestra.run.smithi201.stdout:Depends: linux-image-5.15.0-43-generic, linux-modules-extra-5.15.0-43-generic, linux-firmware, intel-microcode, amd64-microcode 2022-08-05T15:50:39.785 INFO:teuthology.orchestra.run.smithi201.stdout:Recommends: thermald 2022-08-05T15:50:39.785 INFO:teuthology.orchestra.run.smithi201.stdout:Description: Generic Linux kernel image 2022-08-05T15:50:39.786 INFO:teuthology.orchestra.run.smithi201.stdout: This package will always depend on the latest generic kernel image 2022-08-05T15:50:39.786 INFO:teuthology.orchestra.run.smithi201.stdout: available. 2022-08-05T15:50:39.787 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-122-generic Expected: 5.15.0-43-generic 2022-08-05T15:50:39.787 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-08-05T15:50:39.788 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-08-05T15:50:39.788 INFO:teuthology.task.kernel:Installing distro kernel on host.c... 2022-08-05T15:50:39.789 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi201.front.sepia.ceph.com, path=None, version=distro) 2022-08-05T15:50:39.791 DEBUG:teuthology.orchestra.run.smithi201:> sudo apt-get clean 2022-08-05T15:50:39.893 DEBUG:teuthology.orchestra.run.smithi201:> sudo apt-get update 2022-08-05T15:50:40.080 INFO:teuthology.orchestra.run.smithi201.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-08-05T15:50:40.182 INFO:teuthology.orchestra.run.smithi201.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-08-05T15:50:40.263 INFO:teuthology.orchestra.run.smithi201.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-08-05T15:50:40.348 INFO:teuthology.orchestra.run.smithi201.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-08-05T15:50:41.045 INFO:teuthology.orchestra.run.smithi157.stdout:done 2022-08-05T15:50:41.431 DEBUG:teuthology.orchestra.run.smithi157:> dpkg -s linux-image-generic-hwe-20.04 2022-08-05T15:50:41.488 INFO:teuthology.orchestra.run.smithi157.stdout:Package: linux-image-generic-hwe-20.04 2022-08-05T15:50:41.488 INFO:teuthology.orchestra.run.smithi157.stdout:Status: install ok installed 2022-08-05T15:50:41.489 INFO:teuthology.orchestra.run.smithi157.stdout:Priority: optional 2022-08-05T15:50:41.489 INFO:teuthology.orchestra.run.smithi157.stdout:Section: kernel 2022-08-05T15:50:41.494 INFO:teuthology.orchestra.run.smithi157.stdout:Installed-Size: 19 2022-08-05T15:50:41.495 INFO:teuthology.orchestra.run.smithi157.stdout:Maintainer: Ubuntu Kernel Team 2022-08-05T15:50:41.495 INFO:teuthology.orchestra.run.smithi157.stdout:Architecture: amd64 2022-08-05T15:50:41.496 INFO:teuthology.orchestra.run.smithi157.stdout:Source: linux-meta-hwe-5.15 2022-08-05T15:50:41.496 INFO:teuthology.orchestra.run.smithi157.stdout:Version: 5.15.0.43.46~20.04.14 2022-08-05T15:50:41.496 INFO:teuthology.orchestra.run.smithi157.stdout:Provides: virtualbox-guest-modules (= 5.15.0-43), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.4-0ubuntu0.1) 2022-08-05T15:50:41.497 INFO:teuthology.orchestra.run.smithi157.stdout:Depends: linux-image-5.15.0-43-generic, linux-modules-extra-5.15.0-43-generic, linux-firmware, intel-microcode, amd64-microcode 2022-08-05T15:50:41.497 INFO:teuthology.orchestra.run.smithi157.stdout:Recommends: thermald 2022-08-05T15:50:41.543 INFO:teuthology.orchestra.run.smithi157.stdout:Description: Generic Linux kernel image 2022-08-05T15:50:41.543 INFO:teuthology.orchestra.run.smithi157.stdout: This package will always depend on the latest generic kernel image 2022-08-05T15:50:41.544 INFO:teuthology.orchestra.run.smithi157.stdout: available. 2022-08-05T15:50:41.544 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-122-generic Expected: 5.15.0-43-generic 2022-08-05T15:50:41.545 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-08-05T15:50:41.545 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-08-05T15:50:41.545 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2022-08-05T15:50:41.546 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi157.front.sepia.ceph.com, path=None, version=distro) 2022-08-05T15:50:41.546 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-get clean 2022-08-05T15:50:41.638 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-get update 2022-08-05T15:50:41.665 INFO:teuthology.orchestra.run.smithi201.stdout:Reading package lists... 2022-08-05T15:50:41.703 DEBUG:teuthology.orchestra.run.smithi201:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-08-05T15:50:41.774 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-08-05T15:50:41.802 INFO:teuthology.orchestra.run.smithi201.stdout:Reading package lists... 2022-08-05T15:50:41.883 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-08-05T15:50:41.929 INFO:teuthology.orchestra.run.smithi059.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-08-05T15:50:41.934 INFO:teuthology.orchestra.run.smithi059.stdout:Sourcing file `/etc/default/grub' 2022-08-05T15:50:41.944 INFO:teuthology.orchestra.run.smithi059.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-08-05T15:50:41.957 INFO:teuthology.orchestra.run.smithi201.stdout:Building dependency tree... 2022-08-05T15:50:41.957 INFO:teuthology.orchestra.run.smithi201.stdout:Reading state information... 2022-08-05T15:50:41.967 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-08-05T15:50:41.968 INFO:teuthology.orchestra.run.smithi059.stdout:Generating grub configuration file ... 2022-08-05T15:50:42.043 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-08-05T15:50:42.139 INFO:teuthology.orchestra.run.smithi201.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.15.0.43.46~20.04.14). 2022-08-05T15:50:42.139 INFO:teuthology.orchestra.run.smithi201.stdout:0 upgraded, 0 newly installed, 0 to remove and 19 not upgraded. 2022-08-05T15:50:42.140 DEBUG:teuthology.orchestra.run.smithi201:> dpkg -s linux-image-generic-hwe-20.04 2022-08-05T15:50:42.173 INFO:teuthology.orchestra.run.smithi201.stdout:Package: linux-image-generic-hwe-20.04 2022-08-05T15:50:42.174 INFO:teuthology.orchestra.run.smithi201.stdout:Status: install ok installed 2022-08-05T15:50:42.178 INFO:teuthology.orchestra.run.smithi201.stdout:Priority: optional 2022-08-05T15:50:42.179 INFO:teuthology.orchestra.run.smithi201.stdout:Section: kernel 2022-08-05T15:50:42.179 INFO:teuthology.orchestra.run.smithi201.stdout:Installed-Size: 19 2022-08-05T15:50:42.180 INFO:teuthology.orchestra.run.smithi201.stdout:Maintainer: Ubuntu Kernel Team 2022-08-05T15:50:42.182 INFO:teuthology.orchestra.run.smithi201.stdout:Architecture: amd64 2022-08-05T15:50:42.183 INFO:teuthology.orchestra.run.smithi201.stdout:Source: linux-meta-hwe-5.15 2022-08-05T15:50:42.183 INFO:teuthology.orchestra.run.smithi201.stdout:Version: 5.15.0.43.46~20.04.14 2022-08-05T15:50:42.184 INFO:teuthology.orchestra.run.smithi201.stdout:Provides: virtualbox-guest-modules (= 5.15.0-43), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.4-0ubuntu0.1) 2022-08-05T15:50:42.184 INFO:teuthology.orchestra.run.smithi201.stdout:Depends: linux-image-5.15.0-43-generic, linux-modules-extra-5.15.0-43-generic, linux-firmware, intel-microcode, amd64-microcode 2022-08-05T15:50:42.184 INFO:teuthology.orchestra.run.smithi201.stdout:Recommends: thermald 2022-08-05T15:50:42.185 INFO:teuthology.orchestra.run.smithi201.stdout:Description: Generic Linux kernel image 2022-08-05T15:50:42.185 INFO:teuthology.orchestra.run.smithi201.stdout: This package will always depend on the latest generic kernel image 2022-08-05T15:50:42.188 INFO:teuthology.orchestra.run.smithi201.stdout: available. 2022-08-05T15:50:42.189 DEBUG:teuthology.orchestra.run.smithi201:> mktemp 2022-08-05T15:50:42.298 INFO:teuthology.orchestra.run.smithi201.stdout:/tmp/tmp.oV40VD4iEL 2022-08-05T15:50:42.306 DEBUG:teuthology.orchestra.run.smithi201:> sudo cp /boot/grub/grub.cfg /tmp/tmp.oV40VD4iEL 2022-08-05T15:50:42.308 INFO:teuthology.orchestra.run.smithi059.stdout:Found linux image: /boot/vmlinuz-5.15.0-43-generic 2022-08-05T15:50:42.316 INFO:teuthology.orchestra.run.smithi059.stdout:Found initrd image: /boot/initrd.img-5.15.0-43-generic 2022-08-05T15:50:42.372 DEBUG:teuthology.orchestra.run.smithi201:> sudo chmod 0666 /tmp/tmp.oV40VD4iEL 2022-08-05T15:50:42.521 DEBUG:teuthology.orchestra.remote:smithi201:/tmp/tmp.oV40VD4iEL is 9KB 2022-08-05T15:50:42.701 DEBUG:teuthology.orchestra.run.smithi201:> rm -fr /tmp/tmp.oV40VD4iEL 2022-08-05T15:50:42.713 INFO:teuthology.orchestra.run.smithi059.stdout:Found linux image: /boot/vmlinuz-5.4.0-122-generic 2022-08-05T15:50:42.730 INFO:teuthology.orchestra.run.smithi059.stdout:Found initrd image: /boot/initrd.img-5.4.0-122-generic 2022-08-05T15:50:42.733 DEBUG:teuthology.orchestra.run.smithi201:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-08-05T15:50:42.814 DEBUG:teuthology.orchestra.run.smithi201:> set -ex 2022-08-05T15:50:42.815 DEBUG:teuthology.orchestra.run.smithi201:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-08-05T15:50:42.815 DEBUG:teuthology.orchestra.run.smithi201:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-08-05T15:50:42.919 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-43-generic 2022-08-05T15:50:42.920 DEBUG:teuthology.orchestra.run.smithi201:> sudo update-grub 2022-08-05T15:50:43.380 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2022-08-05T15:50:43.396 DEBUG:teuthology.orchestra.run.smithi157:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-08-05T15:50:43.469 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2022-08-05T15:50:43.626 INFO:teuthology.orchestra.run.smithi157.stdout:Building dependency tree... 2022-08-05T15:50:43.631 INFO:teuthology.orchestra.run.smithi157.stdout:Reading state information... 2022-08-05T15:50:43.813 INFO:teuthology.orchestra.run.smithi157.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.15.0.43.46~20.04.14). 2022-08-05T15:50:43.833 INFO:teuthology.orchestra.run.smithi157.stdout:0 upgraded, 0 newly installed, 0 to remove and 19 not upgraded. 2022-08-05T15:50:43.834 DEBUG:teuthology.orchestra.run.smithi157:> dpkg -s linux-image-generic-hwe-20.04 2022-08-05T15:50:43.860 INFO:teuthology.orchestra.run.smithi157.stdout:Package: linux-image-generic-hwe-20.04 2022-08-05T15:50:43.860 INFO:teuthology.orchestra.run.smithi157.stdout:Status: install ok installed 2022-08-05T15:50:43.861 INFO:teuthology.orchestra.run.smithi157.stdout:Priority: optional 2022-08-05T15:50:43.861 INFO:teuthology.orchestra.run.smithi157.stdout:Section: kernel 2022-08-05T15:50:43.861 INFO:teuthology.orchestra.run.smithi157.stdout:Installed-Size: 19 2022-08-05T15:50:43.862 INFO:teuthology.orchestra.run.smithi157.stdout:Maintainer: Ubuntu Kernel Team 2022-08-05T15:50:43.862 INFO:teuthology.orchestra.run.smithi157.stdout:Architecture: amd64 2022-08-05T15:50:43.863 INFO:teuthology.orchestra.run.smithi157.stdout:Source: linux-meta-hwe-5.15 2022-08-05T15:50:43.863 INFO:teuthology.orchestra.run.smithi157.stdout:Version: 5.15.0.43.46~20.04.14 2022-08-05T15:50:43.864 INFO:teuthology.orchestra.run.smithi157.stdout:Provides: virtualbox-guest-modules (= 5.15.0-43), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.4-0ubuntu0.1) 2022-08-05T15:50:43.864 INFO:teuthology.orchestra.run.smithi157.stdout:Depends: linux-image-5.15.0-43-generic, linux-modules-extra-5.15.0-43-generic, linux-firmware, intel-microcode, amd64-microcode 2022-08-05T15:50:43.864 INFO:teuthology.orchestra.run.smithi157.stdout:Recommends: thermald 2022-08-05T15:50:43.865 INFO:teuthology.orchestra.run.smithi157.stdout:Description: Generic Linux kernel image 2022-08-05T15:50:43.865 INFO:teuthology.orchestra.run.smithi157.stdout: This package will always depend on the latest generic kernel image 2022-08-05T15:50:43.866 INFO:teuthology.orchestra.run.smithi157.stdout: available. 2022-08-05T15:50:43.866 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2022-08-05T15:50:43.923 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.iqJs9q1Le9 2022-08-05T15:50:43.924 DEBUG:teuthology.orchestra.run.smithi157:> sudo cp /boot/grub/grub.cfg /tmp/tmp.iqJs9q1Le9 2022-08-05T15:50:43.953 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.iqJs9q1Le9 2022-08-05T15:50:44.138 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.iqJs9q1Le9 is 9KB 2022-08-05T15:50:44.185 INFO:teuthology.orchestra.run.smithi201.stderr:Sourcing file `/etc/default/grub' 2022-08-05T15:50:44.185 INFO:teuthology.orchestra.run.smithi201.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-08-05T15:50:44.186 INFO:teuthology.orchestra.run.smithi201.stderr:Generating grub configuration file ... 2022-08-05T15:50:44.226 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.iqJs9q1Le9 2022-08-05T15:50:44.247 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-08-05T15:50:44.306 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2022-08-05T15:50:44.307 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-08-05T15:50:44.307 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-08-05T15:50:44.343 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-43-generic 2022-08-05T15:50:44.344 DEBUG:teuthology.orchestra.run.smithi157:> sudo update-grub 2022-08-05T15:50:44.406 INFO:teuthology.orchestra.run.smithi201.stderr:Found linux image: /boot/vmlinuz-5.15.0-43-generic 2022-08-05T15:50:44.419 INFO:teuthology.orchestra.run.smithi201.stderr:Found initrd image: /boot/initrd.img-5.15.0-43-generic 2022-08-05T15:50:44.789 INFO:teuthology.orchestra.run.smithi201.stderr:Found linux image: /boot/vmlinuz-5.4.0-122-generic 2022-08-05T15:50:44.802 INFO:teuthology.orchestra.run.smithi201.stderr:Found initrd image: /boot/initrd.img-5.4.0-122-generic 2022-08-05T15:50:45.165 INFO:teuthology.orchestra.run.smithi059.stdout:done 2022-08-05T15:50:45.581 DEBUG:teuthology.orchestra.run.smithi059:> dpkg -s linux-image-generic-hwe-20.04 2022-08-05T15:50:45.614 INFO:teuthology.orchestra.run.smithi059.stdout:Package: linux-image-generic-hwe-20.04 2022-08-05T15:50:45.614 INFO:teuthology.orchestra.run.smithi059.stdout:Status: install ok installed 2022-08-05T15:50:45.615 INFO:teuthology.orchestra.run.smithi059.stdout:Priority: optional 2022-08-05T15:50:45.615 INFO:teuthology.orchestra.run.smithi059.stdout:Section: kernel 2022-08-05T15:50:45.617 INFO:teuthology.orchestra.run.smithi059.stdout:Installed-Size: 19 2022-08-05T15:50:45.617 INFO:teuthology.orchestra.run.smithi059.stdout:Maintainer: Ubuntu Kernel Team 2022-08-05T15:50:45.618 INFO:teuthology.orchestra.run.smithi059.stdout:Architecture: amd64 2022-08-05T15:50:45.618 INFO:teuthology.orchestra.run.smithi059.stdout:Source: linux-meta-hwe-5.15 2022-08-05T15:50:45.619 INFO:teuthology.orchestra.run.smithi059.stdout:Version: 5.15.0.43.46~20.04.14 2022-08-05T15:50:45.619 INFO:teuthology.orchestra.run.smithi059.stdout:Provides: virtualbox-guest-modules (= 5.15.0-43), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.4-0ubuntu0.1) 2022-08-05T15:50:45.627 INFO:teuthology.orchestra.run.smithi059.stdout:Depends: linux-image-5.15.0-43-generic, linux-modules-extra-5.15.0-43-generic, linux-firmware, intel-microcode, amd64-microcode 2022-08-05T15:50:45.627 INFO:teuthology.orchestra.run.smithi059.stdout:Recommends: thermald 2022-08-05T15:50:45.628 INFO:teuthology.orchestra.run.smithi059.stdout:Description: Generic Linux kernel image 2022-08-05T15:50:45.628 INFO:teuthology.orchestra.run.smithi059.stdout: This package will always depend on the latest generic kernel image 2022-08-05T15:50:45.628 INFO:teuthology.orchestra.run.smithi059.stdout: available. 2022-08-05T15:50:45.628 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-122-generic Expected: 5.15.0-43-generic 2022-08-05T15:50:45.629 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-08-05T15:50:45.629 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-08-05T15:50:45.629 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2022-08-05T15:50:45.629 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi059.front.sepia.ceph.com, path=None, version=distro) 2022-08-05T15:50:45.631 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-get clean 2022-08-05T15:50:45.651 INFO:teuthology.orchestra.run.smithi157.stderr:Sourcing file `/etc/default/grub' 2022-08-05T15:50:45.652 INFO:teuthology.orchestra.run.smithi157.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-08-05T15:50:45.676 INFO:teuthology.orchestra.run.smithi157.stderr:Generating grub configuration file ... 2022-08-05T15:50:45.729 INFO:teuthology.orchestra.run.smithi201.stderr:done 2022-08-05T15:50:45.732 DEBUG:teuthology.orchestra.run.smithi201:> sudo shutdown -r now 2022-08-05T15:50:45.737 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt-get update 2022-08-05T15:50:45.896 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-08-05T15:50:45.950 INFO:teuthology.orchestra.run.smithi157.stderr:Found linux image: /boot/vmlinuz-5.15.0-43-generic 2022-08-05T15:50:45.951 INFO:teuthology.orchestra.run.smithi157.stderr:Found initrd image: /boot/initrd.img-5.15.0-43-generic 2022-08-05T15:50:46.008 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-08-05T15:50:46.081 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-08-05T15:50:46.158 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-08-05T15:50:46.306 INFO:teuthology.orchestra.run.smithi157.stderr:Found linux image: /boot/vmlinuz-5.4.0-122-generic 2022-08-05T15:50:46.326 INFO:teuthology.orchestra.run.smithi157.stderr:Found initrd image: /boot/initrd.img-5.4.0-122-generic 2022-08-05T15:50:47.247 INFO:teuthology.orchestra.run.smithi157.stderr:done 2022-08-05T15:50:47.248 DEBUG:teuthology.orchestra.run.smithi157:> sudo shutdown -r now 2022-08-05T15:50:47.506 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2022-08-05T15:50:47.533 DEBUG:teuthology.orchestra.run.smithi059:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic-hwe-20.04 2022-08-05T15:50:47.633 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2022-08-05T15:50:47.791 INFO:teuthology.orchestra.run.smithi059.stdout:Building dependency tree... 2022-08-05T15:50:47.801 INFO:teuthology.orchestra.run.smithi059.stdout:Reading state information... 2022-08-05T15:50:47.994 INFO:teuthology.orchestra.run.smithi059.stdout:linux-image-generic-hwe-20.04 is already the newest version (5.15.0.43.46~20.04.14). 2022-08-05T15:50:47.994 INFO:teuthology.orchestra.run.smithi059.stdout:0 upgraded, 0 newly installed, 0 to remove and 19 not upgraded. 2022-08-05T15:50:47.995 DEBUG:teuthology.orchestra.run.smithi059:> dpkg -s linux-image-generic-hwe-20.04 2022-08-05T15:50:48.080 INFO:teuthology.orchestra.run.smithi059.stdout:Package: linux-image-generic-hwe-20.04 2022-08-05T15:50:48.081 INFO:teuthology.orchestra.run.smithi059.stdout:Status: install ok installed 2022-08-05T15:50:48.081 INFO:teuthology.orchestra.run.smithi059.stdout:Priority: optional 2022-08-05T15:50:48.090 INFO:teuthology.orchestra.run.smithi059.stdout:Section: kernel 2022-08-05T15:50:48.118 INFO:teuthology.orchestra.run.smithi059.stdout:Installed-Size: 19 2022-08-05T15:50:48.119 INFO:teuthology.orchestra.run.smithi059.stdout:Maintainer: Ubuntu Kernel Team 2022-08-05T15:50:48.119 INFO:teuthology.orchestra.run.smithi059.stdout:Architecture: amd64 2022-08-05T15:50:48.120 INFO:teuthology.orchestra.run.smithi059.stdout:Source: linux-meta-hwe-5.15 2022-08-05T15:50:48.120 INFO:teuthology.orchestra.run.smithi059.stdout:Version: 5.15.0.43.46~20.04.14 2022-08-05T15:50:48.120 INFO:teuthology.orchestra.run.smithi059.stdout:Provides: virtualbox-guest-modules (= 5.15.0-43), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.4-0ubuntu0.1) 2022-08-05T15:50:48.121 INFO:teuthology.orchestra.run.smithi059.stdout:Depends: linux-image-5.15.0-43-generic, linux-modules-extra-5.15.0-43-generic, linux-firmware, intel-microcode, amd64-microcode 2022-08-05T15:50:48.121 INFO:teuthology.orchestra.run.smithi059.stdout:Recommends: thermald 2022-08-05T15:50:48.141 INFO:teuthology.orchestra.run.smithi059.stdout:Description: Generic Linux kernel image 2022-08-05T15:50:48.142 INFO:teuthology.orchestra.run.smithi059.stdout: This package will always depend on the latest generic kernel image 2022-08-05T15:50:48.142 INFO:teuthology.orchestra.run.smithi059.stdout: available. 2022-08-05T15:50:48.143 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2022-08-05T15:50:48.200 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.sjBd1ZC8bW 2022-08-05T15:50:48.201 DEBUG:teuthology.orchestra.run.smithi059:> sudo cp /boot/grub/grub.cfg /tmp/tmp.sjBd1ZC8bW 2022-08-05T15:50:48.282 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.sjBd1ZC8bW 2022-08-05T15:50:48.472 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.sjBd1ZC8bW is 9KB 2022-08-05T15:50:48.635 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.sjBd1ZC8bW 2022-08-05T15:50:48.660 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-08-05T15:50:48.714 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-08-05T15:50:48.715 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-08-05T15:50:48.715 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-08-05T15:50:48.753 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-43-generic 2022-08-05T15:50:48.753 DEBUG:teuthology.orchestra.run.smithi059:> sudo update-grub 2022-08-05T15:50:50.084 INFO:teuthology.orchestra.run.smithi059.stderr:Sourcing file `/etc/default/grub' 2022-08-05T15:50:50.085 INFO:teuthology.orchestra.run.smithi059.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-08-05T15:50:50.107 INFO:teuthology.orchestra.run.smithi059.stderr:Generating grub configuration file ... 2022-08-05T15:50:50.348 INFO:teuthology.orchestra.run.smithi059.stderr:Found linux image: /boot/vmlinuz-5.15.0-43-generic 2022-08-05T15:50:50.365 INFO:teuthology.orchestra.run.smithi059.stderr:Found initrd image: /boot/initrd.img-5.15.0-43-generic 2022-08-05T15:50:50.715 INFO:teuthology.orchestra.run.smithi059.stderr:Found linux image: /boot/vmlinuz-5.4.0-122-generic 2022-08-05T15:50:50.731 INFO:teuthology.orchestra.run.smithi059.stderr:Found initrd image: /boot/initrd.img-5.4.0-122-generic 2022-08-05T15:50:52.113 INFO:teuthology.orchestra.run.smithi059.stderr:done 2022-08-05T15:50:52.115 DEBUG:teuthology.orchestra.run.smithi059:> sudo shutdown -r now 2022-08-05T15:51:15.810 INFO:teuthology.task.kernel:Checking client host.c for new kernel version... 2022-08-05T15:51:15.811 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi201.front.sepia.ceph.com' 2022-08-05T15:51:15.813 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi201.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-08-05T15:51:17.261 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2022-08-05T15:51:17.262 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2022-08-05T15:51:17.263 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-08-05T15:51:22.126 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2022-08-05T15:51:22.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2022-08-05T15:51:22.136 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-08-05T15:51:34.139 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.201 2022-08-05T15:51:35.674 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2022-08-05T15:51:40.548 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.59 2022-08-05T15:52:04.171 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi201.front.sepia.ceph.com' 2022-08-05T15:52:04.177 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi201.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-08-05T15:52:05.682 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2022-08-05T15:52:05.683 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-08-05T15:52:07.259 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.201 2022-08-05T15:52:08.766 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2022-08-05T15:52:10.551 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2022-08-05T15:52:10.553 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-08-05T15:52:26.605 DEBUG:teuthology.orchestra.run.smithi059:> true 2022-08-05T15:52:27.315 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2022-08-05T15:52:27.329 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.15.0-43-generic"... 2022-08-05T15:52:27.330 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2022-08-05T15:52:27.367 INFO:teuthology.orchestra.run.smithi059.stdout:5.15.0-43-generic 2022-08-05T15:52:27.368 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-43-generic vs 5.15.0-43-generic 2022-08-05T15:52:27.368 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-08-05T15:52:27.369 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-08-05T15:52:28.371 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2022-08-05T15:52:28.372 DEBUG:teuthology.orchestra.run.smithi059:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-08-05T15:52:28.435 INFO:teuthology.orchestra.run.smithi059.stdout:ttyS1 2022-08-05T15:52:28.463 DEBUG:teuthology.parallel:result is None 2022-08-05T15:52:37.261 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi201.front.sepia.ceph.com' 2022-08-05T15:52:37.262 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi201.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-08-05T15:52:37.720 DEBUG:teuthology.orchestra.run.smithi201:> true 2022-08-05T15:52:38.206 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi201.front.sepia.ceph.com' 2022-08-05T15:52:38.222 INFO:teuthology.task.kernel:Checking kernel version of host.c, want "5.15.0-43-generic"... 2022-08-05T15:52:38.223 DEBUG:teuthology.orchestra.run.smithi201:> uname -r 2022-08-05T15:52:38.266 INFO:teuthology.orchestra.run.smithi201.stdout:5.15.0-43-generic 2022-08-05T15:52:38.267 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-43-generic vs 5.15.0-43-generic 2022-08-05T15:52:38.279 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-08-05T15:52:38.280 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-08-05T15:52:38.770 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2022-08-05T15:52:38.772 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-08-05T15:52:39.282 INFO:teuthology.task.kernel:Enabling kdb on host.c... 2022-08-05T15:52:39.283 DEBUG:teuthology.orchestra.run.smithi201:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-08-05T15:52:39.428 INFO:teuthology.orchestra.run.smithi201.stdout:ttyS1 2022-08-05T15:52:39.446 DEBUG:teuthology.parallel:result is None 2022-08-05T15:52:54.677 DEBUG:teuthology.orchestra.run.smithi157:> true 2022-08-05T15:52:55.210 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2022-08-05T15:52:55.211 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.15.0-43-generic"... 2022-08-05T15:52:55.211 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2022-08-05T15:52:55.274 INFO:teuthology.orchestra.run.smithi157.stdout:5.15.0-43-generic 2022-08-05T15:52:55.275 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-43-generic vs 5.15.0-43-generic 2022-08-05T15:52:55.276 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-08-05T15:52:55.277 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-08-05T15:52:56.281 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2022-08-05T15:52:56.283 DEBUG:teuthology.orchestra.run.smithi157:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-08-05T15:52:56.376 INFO:teuthology.orchestra.run.smithi157.stdout:ttyS1 2022-08-05T15:52:56.432 DEBUG:teuthology.parallel:result is None 2022-08-05T15:52:56.438 INFO:teuthology.run_tasks:Running task internal.base... 2022-08-05T15:52:56.567 INFO:teuthology.task.internal:Creating test directory... 2022-08-05T15:52:56.569 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-08-05T15:52:56.581 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-08-05T15:52:56.611 DEBUG:teuthology.orchestra.run.smithi201:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-08-05T15:52:56.661 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-08-05T15:52:56.784 INFO:teuthology.run_tasks:Running task internal.archive... 2022-08-05T15:52:56.800 INFO:teuthology.task.internal:Creating archive directory... 2022-08-05T15:52:56.801 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-08-05T15:52:56.806 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-08-05T15:52:56.819 DEBUG:teuthology.orchestra.run.smithi201:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-08-05T15:52:56.885 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-08-05T15:52:56.937 INFO:teuthology.task.internal:Enabling coredump saving... 2022-08-05T15:52:56.938 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-08-05T15:52:56.972 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-08-05T15:52:57.002 DEBUG:teuthology.orchestra.run.smithi201:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-08-05T15:52:57.003 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-08-05T15:52:57.009 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-08-05T15:52:57.010 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-08-05T15:52:57.011 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-08-05T15:52:57.054 INFO:teuthology.orchestra.run.smithi201.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-08-05T15:52:57.054 INFO:teuthology.orchestra.run.smithi201.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-08-05T15:52:57.075 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-08-05T15:52:57.084 INFO:teuthology.task.internal:Configuring sudo... 2022-08-05T15:52:57.085 DEBUG:teuthology.orchestra.run.smithi059:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-08-05T15:52:57.103 DEBUG:teuthology.orchestra.run.smithi157:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-08-05T15:52:57.113 DEBUG:teuthology.orchestra.run.smithi201:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-08-05T15:52:57.131 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-08-05T15:52:57.222 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-08-05T15:52:57.222 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-08-05T15:52:57.255 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-08-05T15:52:57.287 DEBUG:teuthology.orchestra.run.smithi201:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-08-05T15:52:57.329 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-08-05T15:52:57.341 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-08-05T15:52:57.381 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-08-05T15:52:57.382 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-08-05T15:52:57.448 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-08-05T15:52:57.460 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-08-05T15:52:57.580 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2022-08-05T15:52:57.581 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-08-05T15:52:57.747 DEBUG:teuthology.orchestra.run.smithi201:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-08-05T15:52:57.762 DEBUG:teuthology.orchestra.run.smithi201:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-08-05T15:52:57.829 DEBUG:teuthology.orchestra.run.smithi201:> set -ex 2022-08-05T15:52:57.829 DEBUG:teuthology.orchestra.run.smithi201:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-08-05T15:52:57.894 DEBUG:teuthology.orchestra.run.smithi059:> sudo service rsyslog restart 2022-08-05T15:52:57.915 DEBUG:teuthology.orchestra.run.smithi157:> sudo service rsyslog restart 2022-08-05T15:52:57.952 DEBUG:teuthology.orchestra.run.smithi201:> sudo service rsyslog restart 2022-08-05T15:52:58.028 INFO:teuthology.run_tasks:Running task internal.timer... 2022-08-05T15:52:58.102 INFO:teuthology.task.internal:Starting timer... 2022-08-05T15:52:58.103 INFO:teuthology.run_tasks:Running task pcp... 2022-08-05T15:52:58.175 INFO:teuthology.run_tasks:Running task selinux... 2022-08-05T15:52:58.235 DEBUG:teuthology.task.selinux:Excluding smithi059: OS 'ubuntu' does not support SELinux 2022-08-05T15:52:58.236 DEBUG:teuthology.task.selinux:Excluding smithi157: OS 'ubuntu' does not support SELinux 2022-08-05T15:52:58.236 DEBUG:teuthology.task.selinux:Excluding smithi201: OS 'ubuntu' does not support SELinux 2022-08-05T15:52:58.237 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-08-05T15:52:58.237 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-08-05T15:52:58.237 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-08-05T15:52:58.238 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-08-05T15:52:58.352 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2022-08-05T15:52:58.353 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main to origin/main 2022-08-05T15:52:58.554 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-08-05T15:52:58.557 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi059.front.sepia.ceph.com,smithi157.front.sepia.ceph.com,smithi201.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main/cephlab.yml 2022-08-05T16:03:34.546 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi059.front.sepia.ceph.com'), Remote(name='ubuntu@smithi157.front.sepia.ceph.com'), Remote(name='ubuntu@smithi201.front.sepia.ceph.com')] 2022-08-05T16:03:34.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2022-08-05T16:03:34.609 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-08-05T16:03:36.351 DEBUG:teuthology.orchestra.run.smithi059:> true 2022-08-05T16:03:36.516 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2022-08-05T16:03:36.609 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-08-05T16:03:38.547 DEBUG:teuthology.orchestra.run.smithi157:> true 2022-08-05T16:03:38.718 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi201.front.sepia.ceph.com' 2022-08-05T16:03:38.779 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi201.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-08-05T16:03:39.810 DEBUG:teuthology.orchestra.run.smithi201:> true 2022-08-05T16:03:39.954 INFO:teuthology.run_tasks:Running task clock... 2022-08-05T16:03:40.221 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-08-05T16:03:40.233 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-08-05T16:03:40.286 DEBUG:teuthology.orchestra.run.smithi059:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-08-05T16:03:40.299 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-08-05T16:03:40.300 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-08-05T16:03:40.351 INFO:teuthology.orchestra.run.smithi059.stdout: 5 Aug 16:03:40 ntpd[16666]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-08-05T16:03:40.352 INFO:teuthology.orchestra.run.smithi059.stdout: 5 Aug 16:03:40 ntpd[16666]: Command line: ntpd -gq 2022-08-05T16:03:40.382 INFO:teuthology.orchestra.run.smithi059.stdout: 5 Aug 16:03:40 ntpd[16666]: proto: precision = 0.085 usec (-23) 2022-08-05T16:03:40.383 INFO:teuthology.orchestra.run.smithi059.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-08-05T16:03:40.384 INFO:teuthology.orchestra.run.smithi059.stdout: 5 Aug 16:03:40 ntpd[16666]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-08-05T16:03:40.385 INFO:teuthology.orchestra.run.smithi059.stderr:restrict ::: KOD does nothing without LIMITED. 2022-08-05T16:03:40.400 INFO:teuthology.orchestra.run.smithi059.stdout: 5 Aug 16:03:40 ntpd[16666]: restrict ::: KOD does nothing without LIMITED. 2022-08-05T16:03:40.400 INFO:teuthology.orchestra.run.smithi059.stdout: 5 Aug 16:03:40 ntpd[16666]: Listen and drop on 0 v6wildcard [::]:123 2022-08-05T16:03:40.401 INFO:teuthology.orchestra.run.smithi059.stdout: 5 Aug 16:03:40 ntpd[16666]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-08-05T16:03:40.401 INFO:teuthology.orchestra.run.smithi059.stdout: 5 Aug 16:03:40 ntpd[16666]: Listen normally on 2 lo 127.0.0.1:123 2022-08-05T16:03:40.402 INFO:teuthology.orchestra.run.smithi059.stdout: 5 Aug 16:03:40 ntpd[16666]: Listen normally on 3 ens1f0 172.21.15.59:123 2022-08-05T16:03:40.402 INFO:teuthology.orchestra.run.smithi059.stdout: 5 Aug 16:03:40 ntpd[16666]: Listen normally on 4 lo [::1]:123 2022-08-05T16:03:40.403 INFO:teuthology.orchestra.run.smithi059.stdout: 5 Aug 16:03:40 ntpd[16666]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:192a%3]:123 2022-08-05T16:03:40.403 INFO:teuthology.orchestra.run.smithi059.stdout: 5 Aug 16:03:40 ntpd[16666]: Listening on routing socket on fd #22 for interface updates 2022-08-05T16:03:40.404 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-08-05T16:03:40.405 DEBUG:teuthology.orchestra.run.smithi201:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-08-05T16:03:40.429 INFO:teuthology.orchestra.run.smithi157.stdout: 5 Aug 16:03:40 ntpd[16790]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-08-05T16:03:40.429 INFO:teuthology.orchestra.run.smithi157.stdout: 5 Aug 16:03:40 ntpd[16790]: Command line: ntpd -gq 2022-08-05T16:03:40.430 INFO:teuthology.orchestra.run.smithi157.stdout: 5 Aug 16:03:40 ntpd[16790]: proto: precision = 0.093 usec (-23) 2022-08-05T16:03:40.481 INFO:teuthology.orchestra.run.smithi157.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-08-05T16:03:40.545 INFO:teuthology.orchestra.run.smithi157.stdout: 5 Aug 16:03:40 ntpd[16790]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-08-05T16:03:40.547 INFO:teuthology.orchestra.run.smithi157.stderr:restrict ::: KOD does nothing without LIMITED. 2022-08-05T16:03:40.550 INFO:teuthology.orchestra.run.smithi157.stdout: 5 Aug 16:03:40 ntpd[16790]: restrict ::: KOD does nothing without LIMITED. 2022-08-05T16:03:40.550 INFO:teuthology.orchestra.run.smithi157.stdout: 5 Aug 16:03:40 ntpd[16790]: Listen and drop on 0 v6wildcard [::]:123 2022-08-05T16:03:40.551 INFO:teuthology.orchestra.run.smithi157.stdout: 5 Aug 16:03:40 ntpd[16790]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-08-05T16:03:40.551 INFO:teuthology.orchestra.run.smithi157.stdout: 5 Aug 16:03:40 ntpd[16790]: Listen normally on 2 lo 127.0.0.1:123 2022-08-05T16:03:40.552 INFO:teuthology.orchestra.run.smithi157.stdout: 5 Aug 16:03:40 ntpd[16790]: Listen normally on 3 enp3s0f1 172.21.15.157:123 2022-08-05T16:03:40.552 INFO:teuthology.orchestra.run.smithi157.stdout: 5 Aug 16:03:40 ntpd[16790]: Listen normally on 4 lo [::1]:123 2022-08-05T16:03:40.553 INFO:teuthology.orchestra.run.smithi157.stdout: 5 Aug 16:03:40 ntpd[16790]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7ea1%5]:123 2022-08-05T16:03:40.556 INFO:teuthology.orchestra.run.smithi157.stdout: 5 Aug 16:03:40 ntpd[16790]: Listening on routing socket on fd #22 for interface updates 2022-08-05T16:03:40.647 INFO:teuthology.orchestra.run.smithi201.stdout: 5 Aug 16:03:40 ntpd[16787]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-08-05T16:03:40.647 INFO:teuthology.orchestra.run.smithi201.stdout: 5 Aug 16:03:40 ntpd[16787]: Command line: ntpd -gq 2022-08-05T16:03:40.648 INFO:teuthology.orchestra.run.smithi201.stdout: 5 Aug 16:03:40 ntpd[16787]: proto: precision = 0.054 usec (-24) 2022-08-05T16:03:40.649 INFO:teuthology.orchestra.run.smithi201.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-08-05T16:03:40.666 INFO:teuthology.orchestra.run.smithi201.stdout: 5 Aug 16:03:40 ntpd[16787]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-08-05T16:03:40.666 INFO:teuthology.orchestra.run.smithi201.stdout: 5 Aug 16:03:40 ntpd[16787]: restrict ::: KOD does nothing without LIMITED. 2022-08-05T16:03:40.668 INFO:teuthology.orchestra.run.smithi201.stderr:restrict ::: KOD does nothing without LIMITED. 2022-08-05T16:03:40.672 INFO:teuthology.orchestra.run.smithi201.stdout: 5 Aug 16:03:40 ntpd[16787]: Listen and drop on 0 v6wildcard [::]:123 2022-08-05T16:03:40.672 INFO:teuthology.orchestra.run.smithi201.stdout: 5 Aug 16:03:40 ntpd[16787]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-08-05T16:03:40.673 INFO:teuthology.orchestra.run.smithi201.stdout: 5 Aug 16:03:40 ntpd[16787]: Listen normally on 2 lo 127.0.0.1:123 2022-08-05T16:03:40.673 INFO:teuthology.orchestra.run.smithi201.stdout: 5 Aug 16:03:40 ntpd[16787]: Listen normally on 3 enp3s0f1 172.21.15.201:123 2022-08-05T16:03:40.738 INFO:teuthology.orchestra.run.smithi201.stdout: 5 Aug 16:03:40 ntpd[16787]: Listen normally on 4 lo [::1]:123 2022-08-05T16:03:40.739 INFO:teuthology.orchestra.run.smithi201.stdout: 5 Aug 16:03:40 ntpd[16787]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7e9d%5]:123 2022-08-05T16:03:40.739 INFO:teuthology.orchestra.run.smithi201.stdout: 5 Aug 16:03:40 ntpd[16787]: Listening on routing socket on fd #22 for interface updates 2022-08-05T16:03:41.327 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:41 ntpd[16666]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:41.328 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:41 ntpd[16666]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:41.391 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:41 ntpd[16790]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:41.392 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:41 ntpd[16790]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:41.392 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:41 ntpd[16790]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:41.392 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:41 ntpd[16790]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:41.393 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:41 ntpd[16790]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:41.393 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:41 ntpd[16790]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:41.623 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:41 ntpd[16787]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:41.624 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:41 ntpd[16787]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:42.387 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:42 ntpd[16790]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:42.388 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:42 ntpd[16790]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:42.399 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:42 ntpd[16790]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:42.400 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:42 ntpd[16790]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:42.400 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:42 ntpd[16790]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:42.401 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:42 ntpd[16790]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:42.401 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:42 ntpd[16790]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:42.401 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:42 ntpd[16790]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:42.596 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:42 ntpd[16787]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:42.597 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:42 ntpd[16787]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:42.597 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:42 ntpd[16787]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:42.603 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:42 ntpd[16787]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:42.604 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:42 ntpd[16787]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:42.604 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:42 ntpd[16787]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:43.361 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:43 ntpd[16666]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:43.362 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:43 ntpd[16666]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:43.362 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:43 ntpd[16666]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:43.364 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:43 ntpd[16666]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:43.364 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:43 ntpd[16666]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:43.364 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:43 ntpd[16666]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:43.465 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:43 ntpd[16790]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:43.465 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:43 ntpd[16790]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:43.471 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:43 ntpd[16790]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:43.471 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:43 ntpd[16790]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:43.472 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:43 ntpd[16790]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:43.472 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:43 ntpd[16790]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:43.593 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:43 ntpd[16787]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:43.594 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:43 ntpd[16787]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:43.594 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:43 ntpd[16787]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:43.594 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:43 ntpd[16787]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:43.595 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:43 ntpd[16787]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:43.595 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:43 ntpd[16787]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:44.375 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:44 ntpd[16790]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:44.376 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:44 ntpd[16790]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:44.376 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:44 ntpd[16790]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:44.376 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:44 ntpd[16790]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:44.376 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:44 ntpd[16790]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:44.376 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:44 ntpd[16790]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:44.377 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:44 ntpd[16790]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:44.377 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:44 ntpd[16790]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:44.377 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:44 ntpd[16790]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:44.377 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:44 ntpd[16790]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:44.401 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:44 ntpd[16790]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:44.401 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:44 ntpd[16790]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:44.631 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:44 ntpd[16787]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:44.632 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:44 ntpd[16787]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:44.632 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:44 ntpd[16787]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:44.633 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:44 ntpd[16787]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:44.633 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:44 ntpd[16787]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:44.633 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:44 ntpd[16787]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:44.648 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:44 ntpd[16787]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:44.649 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:44 ntpd[16787]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:44.649 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:44 ntpd[16787]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:44.649 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:44 ntpd[16787]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:44.649 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:44 ntpd[16787]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:44.649 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:44 ntpd[16787]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:45.397 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:45 ntpd[16666]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:45.397 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:45 ntpd[16666]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:45.417 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:45 ntpd[16666]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:45.418 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:45 ntpd[16666]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:45.418 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:45 ntpd[16666]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:45.421 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:45 ntpd[16666]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:45.422 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:45 ntpd[16666]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:45.422 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:45 ntpd[16666]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:45.423 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:45 ntpd[16666]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:45.423 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:45 ntpd[16666]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:45.423 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:45 ntpd[16666]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:45.424 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:45 ntpd[16666]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:45.424 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:45 ntpd[16666]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:45.425 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:45 ntpd[16666]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:45.454 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:45 ntpd[16666]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:45.455 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:45 ntpd[16666]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:45.455 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:45 ntpd[16666]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:45.456 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:45 ntpd[16666]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:45.587 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:45 ntpd[16790]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:45.640 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:45 ntpd[16790]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:45.640 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:45 ntpd[16790]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:45.686 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:45 ntpd[16790]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:45.687 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:45 ntpd[16790]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:45.687 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:45 ntpd[16790]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:45.689 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:45 ntpd[16787]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:45.717 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:45 ntpd[16787]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:45.727 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:45 ntpd[16787]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:45.727 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:45 ntpd[16787]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:45.728 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:45 ntpd[16787]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:45.728 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:45 ntpd[16787]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:46.404 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:46 ntpd[16790]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:46.415 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:46 ntpd[16790]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:46.415 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:46 ntpd[16790]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:46.415 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:46 ntpd[16790]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:46.421 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:46 ntpd[16790]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:46.422 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:46 ntpd[16790]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:46.423 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:46 ntpd[16790]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:46.423 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:46 ntpd[16790]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:46.423 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:46 ntpd[16790]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:46.423 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:46 ntpd[16790]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:46.424 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:46 ntpd[16790]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:46.425 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:46 ntpd[16790]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:46.613 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:46 ntpd[16787]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:46.613 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:46 ntpd[16787]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:46.614 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:46 ntpd[16787]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:46.648 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:46 ntpd[16787]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:46.648 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:46 ntpd[16787]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:46.709 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:46 ntpd[16787]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:46.709 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:46 ntpd[16787]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:46.733 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:46 ntpd[16787]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:46.733 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:46 ntpd[16787]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:46.733 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:46 ntpd[16787]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:46.742 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:46 ntpd[16787]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:46.764 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:46 ntpd[16787]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:47.402 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:47 ntpd[16666]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:47.403 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:47 ntpd[16666]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:47.403 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:47 ntpd[16666]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:47.404 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:47 ntpd[16666]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:47.404 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:47 ntpd[16666]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:47.405 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:47 ntpd[16666]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:47.405 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:47 ntpd[16666]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:47.405 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:47 ntpd[16666]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:47.406 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:47 ntpd[16666]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:47.406 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:47 ntpd[16666]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:47.436 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:47 ntpd[16666]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:47.436 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:47 ntpd[16666]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:47.436 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:47 ntpd[16666]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:47.436 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:47 ntpd[16666]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:47.437 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:47 ntpd[16666]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:47.437 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:47 ntpd[16666]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:47.437 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:47 ntpd[16666]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:47.437 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:47 ntpd[16666]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:47.439 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:47 ntpd[16790]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:47.439 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:47 ntpd[16790]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:47.439 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:47 ntpd[16790]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:47.439 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:47 ntpd[16790]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:47.440 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:47 ntpd[16790]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:47.440 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:47 ntpd[16790]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:47.440 INFO:teuthology.orchestra.run.smithi157.stdout: 5 Aug 16:03:47 ntpd[16790]: ntpd: time slew +0.000946 s 2022-08-05T16:03:47.478 INFO:teuthology.orchestra.run.smithi157.stdout:ntpd: time slew +0.000946s 2022-08-05T16:03:47.479 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:47 ntpd[16790]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2022-08-05T16:03:47.480 INFO:teuthology.orchestra.run.smithi157.stderr: 5 Aug 16:03:47 ntpd[16790]: can't open /var/log/ntpstats/loopstats.20220805: Permission denied 2022-08-05T16:03:47.487 INFO:teuthology.orchestra.run.smithi157.stdout: remote refid st t when poll reach delay offset jitter 2022-08-05T16:03:47.488 INFO:teuthology.orchestra.run.smithi157.stdout:============================================================================== 2022-08-05T16:03:47.488 INFO:teuthology.orchestra.run.smithi157.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-08-05T16:03:47.494 INFO:teuthology.orchestra.run.smithi157.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-08-05T16:03:47.513 INFO:teuthology.orchestra.run.smithi157.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-08-05T16:03:47.530 INFO:teuthology.orchestra.run.smithi157.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-08-05T16:03:47.605 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:47 ntpd[16787]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:47.605 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:47 ntpd[16787]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:47.616 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:47 ntpd[16787]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:47.617 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:47 ntpd[16787]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:47.617 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:47 ntpd[16787]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:47.618 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:47 ntpd[16787]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:48.631 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:48 ntpd[16787]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:48.631 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:48 ntpd[16787]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:48.632 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:48 ntpd[16787]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:48.632 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:48 ntpd[16787]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:48.632 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:48 ntpd[16787]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:48.632 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:48 ntpd[16787]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:48.633 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:48 ntpd[16787]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:48.633 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:48 ntpd[16787]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:48.633 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:48 ntpd[16787]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:48.633 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:48 ntpd[16787]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:48.633 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:48 ntpd[16787]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:48.643 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:48 ntpd[16787]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:48.644 INFO:teuthology.orchestra.run.smithi201.stdout: 5 Aug 16:03:48 ntpd[16787]: ntpd: time slew +0.000617 s 2022-08-05T16:03:48.644 INFO:teuthology.orchestra.run.smithi201.stdout:ntpd: time slew +0.000617s 2022-08-05T16:03:48.645 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:48 ntpd[16787]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2022-08-05T16:03:48.645 INFO:teuthology.orchestra.run.smithi201.stderr: 5 Aug 16:03:48 ntpd[16787]: can't open /var/log/ntpstats/loopstats.20220805: Permission denied 2022-08-05T16:03:48.667 INFO:teuthology.orchestra.run.smithi201.stdout: remote refid st t when poll reach delay offset jitter 2022-08-05T16:03:48.667 INFO:teuthology.orchestra.run.smithi201.stdout:============================================================================== 2022-08-05T16:03:48.668 INFO:teuthology.orchestra.run.smithi201.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-08-05T16:03:48.668 INFO:teuthology.orchestra.run.smithi201.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-08-05T16:03:48.668 INFO:teuthology.orchestra.run.smithi201.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-08-05T16:03:48.668 INFO:teuthology.orchestra.run.smithi201.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-08-05T16:03:49.338 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:49 ntpd[16666]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:49.339 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:49 ntpd[16666]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:49.339 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:49 ntpd[16666]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:49.339 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:49 ntpd[16666]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:49.339 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:49 ntpd[16666]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:49.340 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:49 ntpd[16666]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:49.340 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:49 ntpd[16666]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:49.340 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:49 ntpd[16666]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:49.340 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:49 ntpd[16666]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:49.340 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:49 ntpd[16666]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:49.341 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:49 ntpd[16666]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-08-05T16:03:49.341 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:49 ntpd[16666]: can't open /var/log/ntpstats/rawstats.20220805: Permission denied 2022-08-05T16:03:49.341 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:49 ntpd[16666]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-08-05T16:03:49.342 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:49 ntpd[16666]: can't open /var/log/ntpstats/peerstats.20220805: Permission denied 2022-08-05T16:03:49.343 INFO:teuthology.orchestra.run.smithi059.stdout: 5 Aug 16:03:49 ntpd[16666]: ntpd: time slew +0.000215 s 2022-08-05T16:03:49.343 INFO:teuthology.orchestra.run.smithi059.stdout:ntpd: time slew +0.000215s 2022-08-05T16:03:49.343 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:49 ntpd[16666]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2022-08-05T16:03:49.344 INFO:teuthology.orchestra.run.smithi059.stderr: 5 Aug 16:03:49 ntpd[16666]: can't open /var/log/ntpstats/loopstats.20220805: Permission denied 2022-08-05T16:03:49.384 INFO:teuthology.orchestra.run.smithi059.stdout: remote refid st t when poll reach delay offset jitter 2022-08-05T16:03:49.385 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================== 2022-08-05T16:03:49.385 INFO:teuthology.orchestra.run.smithi059.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-08-05T16:03:49.385 INFO:teuthology.orchestra.run.smithi059.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-08-05T16:03:49.386 INFO:teuthology.orchestra.run.smithi059.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-08-05T16:03:49.386 INFO:teuthology.orchestra.run.smithi059.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-08-05T16:03:49.388 INFO:teuthology.run_tasks:Running task kubeadm... 2022-08-05T16:03:49.476 INFO:tasks.kubeadm:Kubeadm start 2022-08-05T16:03:49.476 INFO:tasks.kubeadm:Config: {'pod_network': 'flannel', 'version': '1.21'} 2022-08-05T16:03:49.477 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-08-05T16:03:49.582 DEBUG:teuthology.orchestra.run.smithi157:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-08-05T16:03:49.588 INFO:teuthology.orchestra.run.smithi059.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-08-05T16:03:49.588 INFO:teuthology.orchestra.run.smithi059.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-08-05T16:03:49.588 INFO:teuthology.orchestra.run.smithi059.stdout:net.ipv4.ip_forward = 1 2022-08-05T16:03:49.673 DEBUG:teuthology.orchestra.run.smithi201:> sudo modprobe br_netfilter && sudo sysctl net.bridge.bridge-nf-call-ip6tables=1 && sudo sysctl net.bridge.bridge-nf-call-iptables=1 && sudo sysctl net.ipv4.ip_forward=1 && sudo swapoff -a 2022-08-05T16:03:49.675 INFO:teuthology.orchestra.run.smithi157.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-08-05T16:03:49.675 INFO:teuthology.orchestra.run.smithi157.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-08-05T16:03:49.676 INFO:teuthology.orchestra.run.smithi157.stdout:net.ipv4.ip_forward = 1 2022-08-05T16:03:49.716 INFO:teuthology.orchestra.run.smithi201.stdout:net.bridge.bridge-nf-call-ip6tables = 1 2022-08-05T16:03:49.717 INFO:teuthology.orchestra.run.smithi201.stdout:net.bridge.bridge-nf-call-iptables = 1 2022-08-05T16:03:49.723 INFO:teuthology.orchestra.run.smithi201.stdout:net.ipv4.ip_forward = 1 2022-08-05T16:03:49.796 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-08-05T16:03:49.918 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-08-05T16:03:49.965 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-08-05T16:03:49.965 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/docker/daemon.json 2022-08-05T16:03:50.113 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2022-08-05T16:03:50.113 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-08-05T16:03:50.174 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2022-08-05T16:03:50.175 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/docker/daemon.json 2022-08-05T16:03:50.315 DEBUG:teuthology.orchestra.run.smithi201:> set -ex 2022-08-05T16:03:50.315 DEBUG:teuthology.orchestra.run.smithi201:> sudo dd if=/etc/docker/daemon.json of=/dev/stdout 2022-08-05T16:03:50.411 DEBUG:teuthology.orchestra.run.smithi201:> set -ex 2022-08-05T16:03:50.411 DEBUG:teuthology.orchestra.run.smithi201:> sudo dd of=/etc/docker/daemon.json 2022-08-05T16:03:50.737 DEBUG:teuthology.orchestra.run.smithi059:> sudo systemctl restart docker || true 2022-08-05T16:03:50.848 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl restart docker || true 2022-08-05T16:03:50.921 DEBUG:teuthology.orchestra.run.smithi201:> sudo systemctl restart docker || true 2022-08-05T16:03:51.998 INFO:tasks.kubeadm:Identifying pod subnet 2022-08-05T16:03:52.054 INFO:tasks.kubeadm:172.21.15.59 is in 172.21.0.0/20 at pos 3898 2022-08-05T16:03:54.286 INFO:tasks.kubeadm:Pod subnet: 10.249.208.0/21 2022-08-05T16:03:54.287 INFO:tasks.kubeadm:Installing kube{adm,ctl,let} 2022-08-05T16:03:54.288 DEBUG:teuthology.orchestra.run.smithi059:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-08-05T16:03:54.373 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-08-05T16:03:54.401 INFO:teuthology.orchestra.run.smithi059.stderr: 2022-08-05T16:03:54.402 INFO:teuthology.orchestra.run.smithi059.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-08-05T16:03:54.411 INFO:teuthology.orchestra.run.smithi059.stderr: 2022-08-05T16:03:54.413 DEBUG:teuthology.orchestra.run.smithi201:> sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils 2022-08-05T16:03:54.444 INFO:teuthology.orchestra.run.smithi157.stderr: 2022-08-05T16:03:54.445 INFO:teuthology.orchestra.run.smithi157.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-08-05T16:03:54.445 INFO:teuthology.orchestra.run.smithi157.stderr: 2022-08-05T16:03:54.511 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-08-05T16:03:54.522 INFO:teuthology.orchestra.run.smithi201.stderr: 2022-08-05T16:03:54.523 INFO:teuthology.orchestra.run.smithi201.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-08-05T16:03:54.523 INFO:teuthology.orchestra.run.smithi201.stderr: 2022-08-05T16:03:54.524 INFO:teuthology.orchestra.run.smithi059.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-08-05T16:03:54.538 INFO:teuthology.orchestra.run.smithi157.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-08-05T16:03:54.608 INFO:teuthology.orchestra.run.smithi201.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-08-05T16:03:54.608 INFO:teuthology.orchestra.run.smithi059.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-08-05T16:03:54.627 INFO:teuthology.orchestra.run.smithi059.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-08-05T16:03:54.674 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-08-05T16:03:54.730 INFO:teuthology.orchestra.run.smithi201.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-08-05T16:03:54.750 INFO:teuthology.orchestra.run.smithi157.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-08-05T16:03:54.832 INFO:teuthology.orchestra.run.smithi201.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-08-05T16:03:55.090 INFO:teuthology.orchestra.run.smithi157.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-08-05T16:03:55.150 INFO:teuthology.orchestra.run.smithi201.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-08-05T16:03:55.165 INFO:teuthology.orchestra.run.smithi059.stdout:Fetched 336 kB in 1s (531 kB/s) 2022-08-05T16:03:55.380 INFO:teuthology.orchestra.run.smithi157.stdout:Fetched 336 kB in 1s (430 kB/s) 2022-08-05T16:03:55.459 INFO:teuthology.orchestra.run.smithi201.stdout:Fetched 336 kB in 1s (429 kB/s) 2022-08-05T16:03:56.131 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2022-08-05T16:03:56.293 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2022-08-05T16:03:56.294 INFO:teuthology.orchestra.run.smithi059.stdout:Building dependency tree... 2022-08-05T16:03:56.295 INFO:teuthology.orchestra.run.smithi059.stdout:Reading state information... 2022-08-05T16:03:56.330 INFO:teuthology.orchestra.run.smithi201.stdout:Reading package lists... 2022-08-05T16:03:56.380 INFO:teuthology.orchestra.run.smithi059.stdout:19 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-08-05T16:03:56.399 INFO:teuthology.orchestra.run.smithi059.stderr: 2022-08-05T16:03:56.400 INFO:teuthology.orchestra.run.smithi059.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-08-05T16:03:56.402 INFO:teuthology.orchestra.run.smithi059.stderr: 2022-08-05T16:03:56.443 INFO:teuthology.orchestra.run.smithi157.stdout:Building dependency tree... 2022-08-05T16:03:56.450 INFO:teuthology.orchestra.run.smithi157.stdout:Reading state information... 2022-08-05T16:03:56.451 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2022-08-05T16:03:56.520 INFO:teuthology.orchestra.run.smithi157.stdout:19 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-08-05T16:03:56.591 INFO:teuthology.orchestra.run.smithi201.stdout:Building dependency tree... 2022-08-05T16:03:56.592 INFO:teuthology.orchestra.run.smithi201.stdout:Reading state information... 2022-08-05T16:03:56.593 INFO:teuthology.orchestra.run.smithi157.stderr: 2022-08-05T16:03:56.593 INFO:teuthology.orchestra.run.smithi157.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-08-05T16:03:56.593 INFO:teuthology.orchestra.run.smithi157.stderr: 2022-08-05T16:03:56.623 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2022-08-05T16:03:56.624 INFO:teuthology.orchestra.run.smithi201.stdout:19 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-08-05T16:03:56.624 INFO:teuthology.orchestra.run.smithi201.stderr: 2022-08-05T16:03:56.624 INFO:teuthology.orchestra.run.smithi201.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-08-05T16:03:56.625 INFO:teuthology.orchestra.run.smithi201.stderr: 2022-08-05T16:03:56.639 INFO:teuthology.orchestra.run.smithi059.stdout:Building dependency tree... 2022-08-05T16:03:56.639 INFO:teuthology.orchestra.run.smithi059.stdout:Reading state information... 2022-08-05T16:03:56.718 INFO:teuthology.orchestra.run.smithi201.stdout:Reading package lists... 2022-08-05T16:03:56.855 INFO:teuthology.orchestra.run.smithi157.stdout:Building dependency tree... 2022-08-05T16:03:56.856 INFO:teuthology.orchestra.run.smithi157.stdout:Reading state information... 2022-08-05T16:03:56.857 INFO:teuthology.orchestra.run.smithi201.stdout:Building dependency tree... 2022-08-05T16:03:56.866 INFO:teuthology.orchestra.run.smithi201.stdout:Reading state information... 2022-08-05T16:03:56.867 INFO:teuthology.orchestra.run.smithi059.stdout:ca-certificates is already the newest version (20211016~20.04.1). 2022-08-05T16:03:56.867 INFO:teuthology.orchestra.run.smithi059.stdout:curl is already the newest version (7.68.0-1ubuntu2.12). 2022-08-05T16:03:56.876 INFO:teuthology.orchestra.run.smithi059.stdout:The following NEW packages will be installed: 2022-08-05T16:03:56.916 INFO:teuthology.orchestra.run.smithi059.stdout: apt-transport-https 2022-08-05T16:03:56.989 INFO:teuthology.orchestra.run.smithi157.stdout:ca-certificates is already the newest version (20211016~20.04.1). 2022-08-05T16:03:57.023 INFO:teuthology.orchestra.run.smithi157.stdout:curl is already the newest version (7.68.0-1ubuntu2.12). 2022-08-05T16:03:57.030 INFO:teuthology.orchestra.run.smithi157.stdout:The following NEW packages will be installed: 2022-08-05T16:03:57.031 INFO:teuthology.orchestra.run.smithi157.stdout: apt-transport-https 2022-08-05T16:03:57.033 INFO:teuthology.orchestra.run.smithi059.stdout:0 upgraded, 1 newly installed, 0 to remove and 19 not upgraded. 2022-08-05T16:03:57.033 INFO:teuthology.orchestra.run.smithi059.stdout:Need to get 1,704 B of archives. 2022-08-05T16:03:57.110 INFO:teuthology.orchestra.run.smithi059.stdout:After this operation, 162 kB of additional disk space will be used. 2022-08-05T16:03:57.110 INFO:teuthology.orchestra.run.smithi059.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.9 [1,704 B] 2022-08-05T16:03:57.111 INFO:teuthology.orchestra.run.smithi157.stdout:0 upgraded, 1 newly installed, 0 to remove and 19 not upgraded. 2022-08-05T16:03:57.111 INFO:teuthology.orchestra.run.smithi157.stdout:Need to get 1,704 B of archives. 2022-08-05T16:03:57.282 INFO:teuthology.orchestra.run.smithi157.stdout:After this operation, 162 kB of additional disk space will be used. 2022-08-05T16:03:57.283 INFO:teuthology.orchestra.run.smithi157.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.9 [1,704 B] 2022-08-05T16:03:57.283 INFO:teuthology.orchestra.run.smithi201.stdout:ca-certificates is already the newest version (20211016~20.04.1). 2022-08-05T16:03:57.284 INFO:teuthology.orchestra.run.smithi201.stdout:curl is already the newest version (7.68.0-1ubuntu2.12). 2022-08-05T16:03:57.284 INFO:teuthology.orchestra.run.smithi201.stdout:The following NEW packages will be installed: 2022-08-05T16:03:57.285 INFO:teuthology.orchestra.run.smithi201.stdout: apt-transport-https 2022-08-05T16:03:57.339 INFO:teuthology.orchestra.run.smithi201.stdout:0 upgraded, 1 newly installed, 0 to remove and 19 not upgraded. 2022-08-05T16:03:57.339 INFO:teuthology.orchestra.run.smithi201.stdout:Need to get 1,704 B of archives. 2022-08-05T16:03:57.339 INFO:teuthology.orchestra.run.smithi201.stdout:After this operation, 162 kB of additional disk space will be used. 2022-08-05T16:03:57.339 INFO:teuthology.orchestra.run.smithi201.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.9 [1,704 B] 2022-08-05T16:03:57.340 INFO:teuthology.orchestra.run.smithi059.stderr:debconf: unable to initialize frontend: Dialog 2022-08-05T16:03:57.341 INFO:teuthology.orchestra.run.smithi059.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-08-05T16:03:57.341 INFO:teuthology.orchestra.run.smithi059.stderr:debconf: falling back to frontend: Readline 2022-08-05T16:03:57.341 INFO:teuthology.orchestra.run.smithi059.stderr:debconf: unable to initialize frontend: Readline 2022-08-05T16:03:57.341 INFO:teuthology.orchestra.run.smithi059.stderr:debconf: (This frontend requires a controlling tty.) 2022-08-05T16:03:57.341 INFO:teuthology.orchestra.run.smithi059.stderr:debconf: falling back to frontend: Teletype 2022-08-05T16:03:57.363 INFO:teuthology.orchestra.run.smithi059.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-08-05T16:03:57.415 INFO:teuthology.orchestra.run.smithi157.stderr:debconf: unable to initialize frontend: Dialog 2022-08-05T16:03:57.427 INFO:teuthology.orchestra.run.smithi157.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-08-05T16:03:57.427 INFO:teuthology.orchestra.run.smithi157.stderr:debconf: falling back to frontend: Readline 2022-08-05T16:03:57.427 INFO:teuthology.orchestra.run.smithi157.stderr:debconf: unable to initialize frontend: Readline 2022-08-05T16:03:57.428 INFO:teuthology.orchestra.run.smithi157.stderr:debconf: (This frontend requires a controlling tty.) 2022-08-05T16:03:57.438 INFO:teuthology.orchestra.run.smithi157.stderr:debconf: falling back to frontend: Teletype 2022-08-05T16:03:57.439 INFO:teuthology.orchestra.run.smithi157.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-08-05T16:03:57.440 INFO:teuthology.orchestra.run.smithi059.stdout:Fetched 1,704 B in 0s (9,849 B/s) 2022-08-05T16:03:57.441 INFO:teuthology.orchestra.run.smithi157.stdout:Fetched 1,704 B in 0s (35.8 kB/s) 2022-08-05T16:03:57.463 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package apt-transport-https. 2022-08-05T16:03:57.464 INFO:teuthology.orchestra.run.smithi201.stderr:debconf: unable to initialize frontend: Dialog 2022-08-05T16:03:57.465 INFO:teuthology.orchestra.run.smithi201.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-08-05T16:03:57.465 INFO:teuthology.orchestra.run.smithi201.stderr:debconf: falling back to frontend: Readline 2022-08-05T16:03:57.526 INFO:teuthology.orchestra.run.smithi157.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109658 files and directories currently installed.) 2022-08-05T16:03:57.526 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../apt-transport-https_2.0.9_all.deb ... 2022-08-05T16:03:57.526 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking apt-transport-https (2.0.9) ... 2022-08-05T16:03:57.527 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package apt-transport-https. 2022-08-05T16:03:57.528 INFO:teuthology.orchestra.run.smithi059.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109658 files and directories currently installed.) 2022-08-05T16:03:57.528 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../apt-transport-https_2.0.9_all.deb ... 2022-08-05T16:03:57.528 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking apt-transport-https (2.0.9) ... 2022-08-05T16:03:57.529 INFO:teuthology.orchestra.run.smithi201.stderr:debconf: unable to initialize frontend: Readline 2022-08-05T16:03:57.529 INFO:teuthology.orchestra.run.smithi201.stderr:debconf: (This frontend requires a controlling tty.) 2022-08-05T16:03:57.530 INFO:teuthology.orchestra.run.smithi201.stderr:debconf: falling back to frontend: Teletype 2022-08-05T16:03:57.530 INFO:teuthology.orchestra.run.smithi201.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-08-05T16:03:57.530 INFO:teuthology.orchestra.run.smithi201.stdout:Fetched 1,704 B in 0s (9,910 B/s) 2022-08-05T16:03:57.626 INFO:teuthology.orchestra.run.smithi201.stdout:Selecting previously unselected package apt-transport-https. 2022-08-05T16:03:57.648 INFO:teuthology.orchestra.run.smithi201.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109658 files and directories currently installed.) 2022-08-05T16:03:57.648 INFO:teuthology.orchestra.run.smithi201.stdout:Preparing to unpack .../apt-transport-https_2.0.9_all.deb ... 2022-08-05T16:03:57.733 INFO:teuthology.orchestra.run.smithi201.stdout:Unpacking apt-transport-https (2.0.9) ... 2022-08-05T16:03:57.803 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up apt-transport-https (2.0.9) ... 2022-08-05T16:03:57.813 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up apt-transport-https (2.0.9) ... 2022-08-05T16:03:57.979 INFO:teuthology.orchestra.run.smithi201.stdout:Setting up apt-transport-https (2.0.9) ... 2022-08-05T16:03:58.333 INFO:teuthology.orchestra.run.smithi059.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-08-05T16:03:58.356 INFO:teuthology.orchestra.run.smithi059.stderr: 2022-08-05T16:03:58.356 INFO:teuthology.orchestra.run.smithi059.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-08-05T16:03:58.362 INFO:teuthology.orchestra.run.smithi059.stderr: 2022-08-05T16:03:58.364 INFO:teuthology.orchestra.run.smithi157.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-08-05T16:03:58.374 INFO:teuthology.orchestra.run.smithi157.stderr: 2022-08-05T16:03:58.374 INFO:teuthology.orchestra.run.smithi157.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-08-05T16:03:58.374 INFO:teuthology.orchestra.run.smithi157.stderr: 2022-08-05T16:03:58.487 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-08-05T16:03:58.569 INFO:teuthology.orchestra.run.smithi201.stdout:deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main 2022-08-05T16:03:58.570 INFO:teuthology.orchestra.run.smithi201.stderr: 2022-08-05T16:03:58.571 INFO:teuthology.orchestra.run.smithi201.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-08-05T16:03:58.571 INFO:teuthology.orchestra.run.smithi201.stderr: 2022-08-05T16:03:58.572 INFO:teuthology.orchestra.run.smithi157.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-08-05T16:03:58.572 INFO:teuthology.orchestra.run.smithi157.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-08-05T16:03:58.625 INFO:teuthology.orchestra.run.smithi157.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-08-05T16:03:58.625 INFO:teuthology.orchestra.run.smithi059.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-08-05T16:03:58.651 INFO:teuthology.orchestra.run.smithi059.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-08-05T16:03:58.653 INFO:teuthology.orchestra.run.smithi059.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-08-05T16:03:58.773 INFO:teuthology.orchestra.run.smithi157.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-08-05T16:03:58.791 INFO:teuthology.orchestra.run.smithi201.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-08-05T16:03:58.791 INFO:teuthology.orchestra.run.smithi201.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease 2022-08-05T16:03:58.861 INFO:teuthology.orchestra.run.smithi201.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-08-05T16:03:58.958 INFO:teuthology.orchestra.run.smithi201.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-08-05T16:03:58.959 INFO:teuthology.orchestra.run.smithi059.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [9,383 B] 2022-08-05T16:03:58.983 INFO:teuthology.orchestra.run.smithi157.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [57.6 kB] 2022-08-05T16:03:58.993 INFO:teuthology.orchestra.run.smithi059.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-08-05T16:03:59.076 INFO:teuthology.orchestra.run.smithi201.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [57.6 kB] 2022-08-05T16:03:59.188 INFO:teuthology.orchestra.run.smithi201.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-08-05T16:03:59.253 INFO:teuthology.orchestra.run.smithi157.stdout:Fetched 403 kB in 1s (595 kB/s) 2022-08-05T16:03:59.254 INFO:teuthology.orchestra.run.smithi059.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [57.6 kB] 2022-08-05T16:03:59.447 INFO:teuthology.orchestra.run.smithi059.stdout:Fetched 403 kB in 1s (417 kB/s) 2022-08-05T16:03:59.522 INFO:teuthology.orchestra.run.smithi201.stdout:Fetched 403 kB in 1s (469 kB/s) 2022-08-05T16:04:00.128 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2022-08-05T16:04:00.281 INFO:teuthology.orchestra.run.smithi157.stdout:Building dependency tree... 2022-08-05T16:04:00.324 INFO:teuthology.orchestra.run.smithi157.stdout:Reading state information... 2022-08-05T16:04:00.363 INFO:teuthology.orchestra.run.smithi157.stdout:19 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-08-05T16:04:00.399 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2022-08-05T16:04:00.400 INFO:teuthology.orchestra.run.smithi157.stderr: 2022-08-05T16:04:00.400 INFO:teuthology.orchestra.run.smithi157.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-08-05T16:04:00.426 INFO:teuthology.orchestra.run.smithi157.stderr: 2022-08-05T16:04:00.545 INFO:teuthology.orchestra.run.smithi201.stdout:Reading package lists... 2022-08-05T16:04:00.546 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2022-08-05T16:04:00.564 INFO:teuthology.orchestra.run.smithi059.stdout:Building dependency tree... 2022-08-05T16:04:00.565 INFO:teuthology.orchestra.run.smithi059.stdout:Reading state information... 2022-08-05T16:04:00.606 INFO:teuthology.orchestra.run.smithi201.stdout:Building dependency tree... 2022-08-05T16:04:00.606 INFO:teuthology.orchestra.run.smithi201.stdout:Reading state information... 2022-08-05T16:04:00.625 INFO:teuthology.orchestra.run.smithi157.stdout:Building dependency tree... 2022-08-05T16:04:00.668 INFO:teuthology.orchestra.run.smithi157.stdout:Reading state information... 2022-08-05T16:04:00.697 INFO:teuthology.orchestra.run.smithi059.stdout:19 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-08-05T16:04:00.698 INFO:teuthology.orchestra.run.smithi059.stderr: 2022-08-05T16:04:00.698 INFO:teuthology.orchestra.run.smithi059.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-08-05T16:04:00.699 INFO:teuthology.orchestra.run.smithi059.stderr: 2022-08-05T16:04:00.700 INFO:teuthology.orchestra.run.smithi201.stdout:19 packages can be upgraded. Run 'apt list --upgradable' to see them. 2022-08-05T16:04:00.726 INFO:teuthology.orchestra.run.smithi201.stderr: 2022-08-05T16:04:00.727 INFO:teuthology.orchestra.run.smithi201.stderr:WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 2022-08-05T16:04:00.799 INFO:teuthology.orchestra.run.smithi201.stderr: 2022-08-05T16:04:00.908 INFO:teuthology.orchestra.run.smithi059.stdout:Reading package lists... 2022-08-05T16:04:00.933 INFO:teuthology.orchestra.run.smithi201.stdout:Reading package lists... 2022-08-05T16:04:00.963 INFO:teuthology.orchestra.run.smithi157.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-08-05T16:04:00.964 INFO:teuthology.orchestra.run.smithi157.stdout:bridge-utils set to manually installed. 2022-08-05T16:04:00.964 INFO:teuthology.orchestra.run.smithi157.stdout:The following additional packages will be installed: 2022-08-05T16:04:00.965 INFO:teuthology.orchestra.run.smithi157.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-08-05T16:04:00.965 INFO:teuthology.orchestra.run.smithi157.stdout:Suggested packages: 2022-08-05T16:04:00.965 INFO:teuthology.orchestra.run.smithi157.stdout: nftables 2022-08-05T16:04:01.038 INFO:teuthology.orchestra.run.smithi157.stdout:The following NEW packages will be installed: 2022-08-05T16:04:01.054 INFO:teuthology.orchestra.run.smithi157.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-08-05T16:04:01.055 INFO:teuthology.orchestra.run.smithi157.stdout:0 upgraded, 8 newly installed, 0 to remove and 19 not upgraded. 2022-08-05T16:04:01.055 INFO:teuthology.orchestra.run.smithi157.stdout:Need to get 75.2 MB of archives. 2022-08-05T16:04:01.055 INFO:teuthology.orchestra.run.smithi157.stdout:After this operation, 313 MB of additional disk space will be used. 2022-08-05T16:04:01.055 INFO:teuthology.orchestra.run.smithi157.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-08-05T16:04:01.056 INFO:teuthology.orchestra.run.smithi157.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-08-05T16:04:01.072 INFO:teuthology.orchestra.run.smithi157.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-08-05T16:04:01.072 INFO:teuthology.orchestra.run.smithi157.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.24.2-00 [12.3 MB] 2022-08-05T16:04:01.078 INFO:teuthology.orchestra.run.smithi201.stdout:Building dependency tree... 2022-08-05T16:04:01.078 INFO:teuthology.orchestra.run.smithi201.stdout:Reading state information... 2022-08-05T16:04:01.079 INFO:teuthology.orchestra.run.smithi059.stdout:Building dependency tree... 2022-08-05T16:04:01.079 INFO:teuthology.orchestra.run.smithi059.stdout:Reading state information... 2022-08-05T16:04:01.080 INFO:teuthology.orchestra.run.smithi059.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-08-05T16:04:01.080 INFO:teuthology.orchestra.run.smithi059.stdout:bridge-utils set to manually installed. 2022-08-05T16:04:01.080 INFO:teuthology.orchestra.run.smithi059.stdout:The following additional packages will be installed: 2022-08-05T16:04:01.080 INFO:teuthology.orchestra.run.smithi059.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-08-05T16:04:01.081 INFO:teuthology.orchestra.run.smithi059.stdout:Suggested packages: 2022-08-05T16:04:01.081 INFO:teuthology.orchestra.run.smithi059.stdout: nftables 2022-08-05T16:04:01.113 INFO:teuthology.orchestra.run.smithi201.stdout:bridge-utils is already the newest version (1.6-2ubuntu1). 2022-08-05T16:04:01.114 INFO:teuthology.orchestra.run.smithi201.stdout:bridge-utils set to manually installed. 2022-08-05T16:04:01.128 INFO:teuthology.orchestra.run.smithi201.stdout:The following additional packages will be installed: 2022-08-05T16:04:01.128 INFO:teuthology.orchestra.run.smithi201.stdout: conntrack cri-tools ebtables kubernetes-cni socat 2022-08-05T16:04:01.129 INFO:teuthology.orchestra.run.smithi201.stdout:Suggested packages: 2022-08-05T16:04:01.130 INFO:teuthology.orchestra.run.smithi201.stdout: nftables 2022-08-05T16:04:01.131 INFO:teuthology.orchestra.run.smithi059.stdout:The following NEW packages will be installed: 2022-08-05T16:04:01.131 INFO:teuthology.orchestra.run.smithi059.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-08-05T16:04:01.182 INFO:teuthology.orchestra.run.smithi201.stdout:The following NEW packages will be installed: 2022-08-05T16:04:01.183 INFO:teuthology.orchestra.run.smithi201.stdout: conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat 2022-08-05T16:04:01.296 INFO:teuthology.orchestra.run.smithi059.stdout:0 upgraded, 8 newly installed, 0 to remove and 19 not upgraded. 2022-08-05T16:04:01.297 INFO:teuthology.orchestra.run.smithi059.stdout:Need to get 75.2 MB of archives. 2022-08-05T16:04:01.297 INFO:teuthology.orchestra.run.smithi059.stdout:After this operation, 313 MB of additional disk space will be used. 2022-08-05T16:04:01.298 INFO:teuthology.orchestra.run.smithi059.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-08-05T16:04:01.328 INFO:teuthology.orchestra.run.smithi201.stdout:0 upgraded, 8 newly installed, 0 to remove and 19 not upgraded. 2022-08-05T16:04:01.329 INFO:teuthology.orchestra.run.smithi201.stdout:Need to get 75.2 MB of archives. 2022-08-05T16:04:01.329 INFO:teuthology.orchestra.run.smithi201.stdout:After this operation, 313 MB of additional disk space will be used. 2022-08-05T16:04:01.329 INFO:teuthology.orchestra.run.smithi201.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 conntrack amd64 1:1.4.5-2 [30.3 kB] 2022-08-05T16:04:01.355 INFO:teuthology.orchestra.run.smithi157.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-08-05T16:04:01.368 INFO:teuthology.orchestra.run.smithi201.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.24.2-00 [12.3 MB] 2022-08-05T16:04:01.411 INFO:teuthology.orchestra.run.smithi059.stdout:Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 cri-tools amd64 1.24.2-00 [12.3 MB] 2022-08-05T16:04:01.483 INFO:teuthology.orchestra.run.smithi059.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-08-05T16:04:01.548 INFO:teuthology.orchestra.run.smithi201.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 ebtables amd64 2.0.11-3build1 [80.3 kB] 2022-08-05T16:04:01.571 INFO:teuthology.orchestra.run.smithi201.stdout:Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-08-05T16:04:01.640 INFO:teuthology.orchestra.run.smithi059.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-08-05T16:04:01.675 INFO:teuthology.orchestra.run.smithi201.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-08-05T16:04:01.953 INFO:teuthology.orchestra.run.smithi201.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.24.3-00 [19.2 MB] 2022-08-05T16:04:01.979 INFO:teuthology.orchestra.run.smithi157.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.24.3-00 [19.2 MB] 2022-08-05T16:04:02.199 INFO:teuthology.orchestra.run.smithi059.stdout:Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB] 2022-08-05T16:04:02.256 INFO:teuthology.orchestra.run.smithi201.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.24.3-00 [9,316 kB] 2022-08-05T16:04:02.348 INFO:teuthology.orchestra.run.smithi201.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.24.3-00 [9,002 kB] 2022-08-05T16:04:02.471 INFO:teuthology.orchestra.run.smithi157.stdout:Get:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.24.3-00 [9,316 kB] 2022-08-05T16:04:02.710 INFO:teuthology.orchestra.run.smithi157.stdout:Get:8 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.24.3-00 [9,002 kB] 2022-08-05T16:04:02.775 INFO:teuthology.orchestra.run.smithi201.stderr:debconf: unable to initialize frontend: Dialog 2022-08-05T16:04:02.775 INFO:teuthology.orchestra.run.smithi201.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-08-05T16:04:02.776 INFO:teuthology.orchestra.run.smithi201.stderr:debconf: falling back to frontend: Readline 2022-08-05T16:04:02.785 INFO:teuthology.orchestra.run.smithi201.stderr:debconf: unable to initialize frontend: Readline 2022-08-05T16:04:02.786 INFO:teuthology.orchestra.run.smithi201.stderr:debconf: (This frontend requires a controlling tty.) 2022-08-05T16:04:02.787 INFO:teuthology.orchestra.run.smithi201.stderr:debconf: falling back to frontend: Teletype 2022-08-05T16:04:02.788 INFO:teuthology.orchestra.run.smithi201.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-08-05T16:04:02.840 INFO:teuthology.orchestra.run.smithi201.stdout:Fetched 75.2 MB in 1s (53.4 MB/s) 2022-08-05T16:04:02.915 INFO:teuthology.orchestra.run.smithi201.stdout:Selecting previously unselected package conntrack. 2022-08-05T16:04:02.996 INFO:teuthology.orchestra.run.smithi201.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109662 files and directories currently installed.) 2022-08-05T16:04:02.997 INFO:teuthology.orchestra.run.smithi201.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-08-05T16:04:03.017 INFO:teuthology.orchestra.run.smithi201.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-08-05T16:04:03.256 INFO:teuthology.orchestra.run.smithi157.stderr:debconf: unable to initialize frontend: Dialog 2022-08-05T16:04:03.257 INFO:teuthology.orchestra.run.smithi157.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-08-05T16:04:03.257 INFO:teuthology.orchestra.run.smithi157.stderr:debconf: falling back to frontend: Readline 2022-08-05T16:04:03.296 INFO:teuthology.orchestra.run.smithi157.stderr:debconf: unable to initialize frontend: Readline 2022-08-05T16:04:03.296 INFO:teuthology.orchestra.run.smithi157.stderr:debconf: (This frontend requires a controlling tty.) 2022-08-05T16:04:03.297 INFO:teuthology.orchestra.run.smithi157.stderr:debconf: falling back to frontend: Teletype 2022-08-05T16:04:03.297 INFO:teuthology.orchestra.run.smithi157.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-08-05T16:04:03.433 INFO:teuthology.orchestra.run.smithi201.stdout:Selecting previously unselected package cri-tools. 2022-08-05T16:04:03.433 INFO:teuthology.orchestra.run.smithi201.stdout:Preparing to unpack .../1-cri-tools_1.24.2-00_amd64.deb ... 2022-08-05T16:04:03.433 INFO:teuthology.orchestra.run.smithi201.stdout:Unpacking cri-tools (1.24.2-00) ... 2022-08-05T16:04:03.531 INFO:teuthology.orchestra.run.smithi157.stdout:Fetched 75.2 MB in 2s (34.3 MB/s) 2022-08-05T16:04:03.532 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package conntrack. 2022-08-05T16:04:03.532 INFO:teuthology.orchestra.run.smithi157.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109662 files and directories currently installed.) 2022-08-05T16:04:03.533 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-08-05T16:04:03.650 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-08-05T16:04:03.651 INFO:teuthology.orchestra.run.smithi059.stdout:Get:4 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.24.3-00 [19.2 MB] 2022-08-05T16:04:03.903 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package cri-tools. 2022-08-05T16:04:03.903 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../1-cri-tools_1.24.2-00_amd64.deb ... 2022-08-05T16:04:03.937 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking cri-tools (1.24.2-00) ... 2022-08-05T16:04:04.088 INFO:teuthology.orchestra.run.smithi059.stdout:Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.24.3-00 [9,316 kB] 2022-08-05T16:04:04.320 INFO:teuthology.orchestra.run.smithi059.stdout:Get:6 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.24.3-00 [9,002 kB] 2022-08-05T16:04:04.526 INFO:teuthology.orchestra.run.smithi201.stdout:Selecting previously unselected package ebtables. 2022-08-05T16:04:04.527 INFO:teuthology.orchestra.run.smithi201.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-08-05T16:04:04.620 INFO:teuthology.orchestra.run.smithi201.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-08-05T16:04:04.713 INFO:teuthology.orchestra.run.smithi059.stderr:debconf: unable to initialize frontend: Dialog 2022-08-05T16:04:04.713 INFO:teuthology.orchestra.run.smithi059.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2022-08-05T16:04:04.725 INFO:teuthology.orchestra.run.smithi059.stderr:debconf: falling back to frontend: Readline 2022-08-05T16:04:04.725 INFO:teuthology.orchestra.run.smithi059.stderr:debconf: unable to initialize frontend: Readline 2022-08-05T16:04:04.726 INFO:teuthology.orchestra.run.smithi059.stderr:debconf: (This frontend requires a controlling tty.) 2022-08-05T16:04:04.727 INFO:teuthology.orchestra.run.smithi059.stderr:debconf: falling back to frontend: Teletype 2022-08-05T16:04:04.736 INFO:teuthology.orchestra.run.smithi059.stderr:dpkg-preconfigure: unable to re-open stdin: 2022-08-05T16:04:04.760 INFO:teuthology.orchestra.run.smithi059.stdout:Fetched 75.2 MB in 3s (22.4 MB/s) 2022-08-05T16:04:04.821 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package conntrack. 2022-08-05T16:04:04.889 INFO:teuthology.orchestra.run.smithi059.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109662 files and directories currently installed.) 2022-08-05T16:04:04.890 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../0-conntrack_1%3a1.4.5-2_amd64.deb ... 2022-08-05T16:04:04.892 INFO:teuthology.orchestra.run.smithi201.stdout:Selecting previously unselected package kubernetes-cni. 2022-08-05T16:04:04.898 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking conntrack (1:1.4.5-2) ... 2022-08-05T16:04:04.899 INFO:teuthology.orchestra.run.smithi201.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-08-05T16:04:04.955 INFO:teuthology.orchestra.run.smithi201.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-08-05T16:04:05.120 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ebtables. 2022-08-05T16:04:05.133 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-08-05T16:04:05.220 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package cri-tools. 2022-08-05T16:04:05.221 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../1-cri-tools_1.24.2-00_amd64.deb ... 2022-08-05T16:04:05.232 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-08-05T16:04:05.296 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking cri-tools (1.24.2-00) ... 2022-08-05T16:04:05.537 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package kubernetes-cni. 2022-08-05T16:04:05.544 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-08-05T16:04:05.582 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-08-05T16:04:06.387 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package ebtables. 2022-08-05T16:04:06.408 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../2-ebtables_2.0.11-3build1_amd64.deb ... 2022-08-05T16:04:06.419 INFO:teuthology.orchestra.run.smithi201.stdout:Selecting previously unselected package socat. 2022-08-05T16:04:06.434 INFO:teuthology.orchestra.run.smithi201.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-08-05T16:04:06.471 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking ebtables (2.0.11-3build1) ... 2022-08-05T16:04:06.518 INFO:teuthology.orchestra.run.smithi201.stdout:Unpacking socat (1.7.3.3-2) ... 2022-08-05T16:04:06.710 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package kubernetes-cni. 2022-08-05T16:04:06.725 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../3-kubernetes-cni_0.8.7-00_amd64.deb ... 2022-08-05T16:04:06.758 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking kubernetes-cni (0.8.7-00) ... 2022-08-05T16:04:06.894 INFO:teuthology.orchestra.run.smithi201.stdout:Selecting previously unselected package kubelet. 2022-08-05T16:04:06.924 INFO:teuthology.orchestra.run.smithi201.stdout:Preparing to unpack .../5-kubelet_1.24.3-00_amd64.deb ... 2022-08-05T16:04:06.951 INFO:teuthology.orchestra.run.smithi201.stdout:Unpacking kubelet (1.24.3-00) ... 2022-08-05T16:04:07.070 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package socat. 2022-08-05T16:04:07.096 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-08-05T16:04:07.129 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking socat (1.7.3.3-2) ... 2022-08-05T16:04:07.555 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package kubelet. 2022-08-05T16:04:07.570 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../5-kubelet_1.24.3-00_amd64.deb ... 2022-08-05T16:04:07.633 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking kubelet (1.24.3-00) ... 2022-08-05T16:04:08.248 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package socat. 2022-08-05T16:04:08.249 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../4-socat_1.7.3.3-2_amd64.deb ... 2022-08-05T16:04:08.304 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking socat (1.7.3.3-2) ... 2022-08-05T16:04:08.703 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package kubelet. 2022-08-05T16:04:08.704 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../5-kubelet_1.24.3-00_amd64.deb ... 2022-08-05T16:04:08.704 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking kubelet (1.24.3-00) ... 2022-08-05T16:04:09.292 INFO:teuthology.orchestra.run.smithi201.stdout:Selecting previously unselected package kubectl. 2022-08-05T16:04:09.293 INFO:teuthology.orchestra.run.smithi201.stdout:Preparing to unpack .../6-kubectl_1.24.3-00_amd64.deb ... 2022-08-05T16:04:09.330 INFO:teuthology.orchestra.run.smithi201.stdout:Unpacking kubectl (1.24.3-00) ... 2022-08-05T16:04:09.995 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package kubectl. 2022-08-05T16:04:09.995 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../6-kubectl_1.24.3-00_amd64.deb ... 2022-08-05T16:04:10.114 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking kubectl (1.24.3-00) ... 2022-08-05T16:04:10.582 INFO:teuthology.orchestra.run.smithi201.stdout:Selecting previously unselected package kubeadm. 2022-08-05T16:04:10.584 INFO:teuthology.orchestra.run.smithi201.stdout:Preparing to unpack .../7-kubeadm_1.24.3-00_amd64.deb ... 2022-08-05T16:04:10.639 INFO:teuthology.orchestra.run.smithi201.stdout:Unpacking kubeadm (1.24.3-00) ... 2022-08-05T16:04:10.980 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package kubectl. 2022-08-05T16:04:10.999 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../6-kubectl_1.24.3-00_amd64.deb ... 2022-08-05T16:04:11.015 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking kubectl (1.24.3-00) ... 2022-08-05T16:04:11.303 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package kubeadm. 2022-08-05T16:04:11.326 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../7-kubeadm_1.24.3-00_amd64.deb ... 2022-08-05T16:04:11.328 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking kubeadm (1.24.3-00) ... 2022-08-05T16:04:11.870 INFO:teuthology.orchestra.run.smithi201.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-08-05T16:04:11.950 INFO:teuthology.orchestra.run.smithi201.stdout:Setting up kubectl (1.24.3-00) ... 2022-08-05T16:04:12.068 INFO:teuthology.orchestra.run.smithi201.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-08-05T16:04:12.259 INFO:teuthology.orchestra.run.smithi059.stdout:Selecting previously unselected package kubeadm. 2022-08-05T16:04:12.260 INFO:teuthology.orchestra.run.smithi059.stdout:Preparing to unpack .../7-kubeadm_1.24.3-00_amd64.deb ... 2022-08-05T16:04:12.304 INFO:teuthology.orchestra.run.smithi059.stdout:Unpacking kubeadm (1.24.3-00) ... 2022-08-05T16:04:12.526 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-08-05T16:04:12.669 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up kubectl (1.24.3-00) ... 2022-08-05T16:04:12.832 INFO:teuthology.orchestra.run.smithi201.stdout:Setting up socat (1.7.3.3-2) ... 2022-08-05T16:04:12.862 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-08-05T16:04:12.954 INFO:teuthology.orchestra.run.smithi201.stdout:Setting up cri-tools (1.24.2-00) ... 2022-08-05T16:04:13.118 INFO:teuthology.orchestra.run.smithi201.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-08-05T16:04:13.234 INFO:teuthology.orchestra.run.smithi201.stdout:Setting up kubelet (1.24.3-00) ... 2022-08-05T16:04:13.446 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up conntrack (1:1.4.5-2) ... 2022-08-05T16:04:13.457 INFO:teuthology.orchestra.run.smithi201.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-08-05T16:04:13.560 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up kubectl (1.24.3-00) ... 2022-08-05T16:04:13.593 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up socat (1.7.3.3-2) ... 2022-08-05T16:04:13.694 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up cri-tools (1.24.2-00) ... 2022-08-05T16:04:13.704 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up ebtables (2.0.11-3build1) ... 2022-08-05T16:04:13.871 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-08-05T16:04:13.932 INFO:teuthology.orchestra.run.smithi201.stdout:Setting up kubeadm (1.24.3-00) ... 2022-08-05T16:04:13.977 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up kubelet (1.24.3-00) ... 2022-08-05T16:04:14.234 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-08-05T16:04:14.339 INFO:teuthology.orchestra.run.smithi201.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-08-05T16:04:14.435 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up socat (1.7.3.3-2) ... 2022-08-05T16:04:14.572 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up cri-tools (1.24.2-00) ... 2022-08-05T16:04:14.703 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up kubeadm (1.24.3-00) ... 2022-08-05T16:04:14.800 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up kubernetes-cni (0.8.7-00) ... 2022-08-05T16:04:14.942 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up kubelet (1.24.3-00) ... 2022-08-05T16:04:15.119 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-08-05T16:04:15.164 INFO:teuthology.orchestra.run.smithi059.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service -> /lib/systemd/system/kubelet.service. 2022-08-05T16:04:15.622 INFO:teuthology.orchestra.run.smithi059.stdout:Setting up kubeadm (1.24.3-00) ... 2022-08-05T16:04:16.077 INFO:teuthology.orchestra.run.smithi059.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-08-05T16:04:16.986 DEBUG:teuthology.orchestra.run.smithi059:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-08-05T16:04:17.012 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-08-05T16:04:17.040 DEBUG:teuthology.orchestra.run.smithi201:> sudo systemctl enable --now kubelet && sudo kubeadm config images pull 2022-08-05T16:04:28.793 INFO:teuthology.orchestra.run.smithi201.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.24.3 2022-08-05T16:04:29.788 INFO:teuthology.orchestra.run.smithi157.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.24.3 2022-08-05T16:04:30.418 INFO:teuthology.orchestra.run.smithi059.stdout:[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.24.3 2022-08-05T16:04:31.786 INFO:teuthology.orchestra.run.smithi201.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.24.3 2022-08-05T16:04:33.156 INFO:teuthology.orchestra.run.smithi059.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.24.3 2022-08-05T16:04:33.703 INFO:teuthology.orchestra.run.smithi157.stdout:[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.24.3 2022-08-05T16:04:34.521 INFO:teuthology.orchestra.run.smithi201.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.24.3 2022-08-05T16:04:35.524 INFO:teuthology.orchestra.run.smithi059.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.24.3 2022-08-05T16:04:36.096 INFO:teuthology.orchestra.run.smithi157.stdout:[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.24.3 2022-08-05T16:04:38.295 INFO:teuthology.orchestra.run.smithi201.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.24.3 2022-08-05T16:04:39.031 INFO:teuthology.orchestra.run.smithi059.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.24.3 2022-08-05T16:04:39.933 INFO:teuthology.orchestra.run.smithi201.stdout:[config/images] Pulled k8s.gcr.io/pause:3.7 2022-08-05T16:04:39.966 INFO:teuthology.orchestra.run.smithi157.stdout:[config/images] Pulled k8s.gcr.io/kube-proxy:v1.24.3 2022-08-05T16:04:40.818 INFO:teuthology.orchestra.run.smithi059.stdout:[config/images] Pulled k8s.gcr.io/pause:3.7 2022-08-05T16:04:41.831 INFO:teuthology.orchestra.run.smithi157.stdout:[config/images] Pulled k8s.gcr.io/pause:3.7 2022-08-05T16:04:47.321 INFO:teuthology.orchestra.run.smithi059.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.3-0 2022-08-05T16:04:48.964 INFO:teuthology.orchestra.run.smithi201.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.3-0 2022-08-05T16:04:49.952 INFO:teuthology.orchestra.run.smithi059.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-08-05T16:04:51.354 INFO:teuthology.orchestra.run.smithi201.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-08-05T16:04:53.390 INFO:teuthology.orchestra.run.smithi157.stdout:[config/images] Pulled k8s.gcr.io/etcd:3.5.3-0 2022-08-05T16:04:55.899 INFO:teuthology.orchestra.run.smithi157.stdout:[config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6 2022-08-05T16:04:55.918 INFO:tasks.kubeadm:Token: abcdef.6zedpdac7qr7of2m 2022-08-05T16:04:55.919 INFO:tasks.kubeadm:Remotes: {Remote(name='ubuntu@smithi059.front.sepia.ceph.com'): '172.21.15.59', Remote(name='ubuntu@smithi157.front.sepia.ceph.com'): '172.21.15.157', Remote(name='ubuntu@smithi201.front.sepia.ceph.com'): '172.21.15.201'} 2022-08-05T16:04:55.939 DEBUG:teuthology.orchestra.run.smithi059:> sudo kubeadm init --node-name smithi059 --token abcdef.6zedpdac7qr7of2m --pod-network-cidr 10.249.208.0/21 2022-08-05T16:04:56.157 INFO:teuthology.orchestra.run.smithi059.stdout:[init] Using Kubernetes version: v1.24.3 2022-08-05T16:04:56.171 INFO:teuthology.orchestra.run.smithi059.stdout:[preflight] Running pre-flight checks 2022-08-05T16:04:56.317 INFO:teuthology.orchestra.run.smithi059.stdout:[preflight] Pulling images required for setting up a Kubernetes cluster 2022-08-05T16:04:56.318 INFO:teuthology.orchestra.run.smithi059.stdout:[preflight] This might take a minute or two, depending on the speed of your internet connection 2022-08-05T16:04:56.318 INFO:teuthology.orchestra.run.smithi059.stdout:[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 2022-08-05T16:04:56.454 INFO:teuthology.orchestra.run.smithi059.stdout:[certs] Using certificateDir folder "/etc/kubernetes/pki" 2022-08-05T16:04:56.712 INFO:teuthology.orchestra.run.smithi059.stdout:[certs] Generating "ca" certificate and key 2022-08-05T16:04:56.811 INFO:teuthology.orchestra.run.smithi059.stdout:[certs] Generating "apiserver" certificate and key 2022-08-05T16:04:56.811 INFO:teuthology.orchestra.run.smithi059.stdout:[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local smithi059] and IPs [10.96.0.1 172.21.15.59] 2022-08-05T16:04:57.007 INFO:teuthology.orchestra.run.smithi059.stdout:[certs] Generating "apiserver-kubelet-client" certificate and key 2022-08-05T16:04:57.098 INFO:teuthology.orchestra.run.smithi059.stdout:[certs] Generating "front-proxy-ca" certificate and key 2022-08-05T16:04:57.182 INFO:teuthology.orchestra.run.smithi059.stdout:[certs] Generating "front-proxy-client" certificate and key 2022-08-05T16:04:57.349 INFO:teuthology.orchestra.run.smithi059.stdout:[certs] Generating "etcd/ca" certificate and key 2022-08-05T16:04:57.495 INFO:teuthology.orchestra.run.smithi059.stdout:[certs] Generating "etcd/server" certificate and key 2022-08-05T16:04:57.495 INFO:teuthology.orchestra.run.smithi059.stdout:[certs] etcd/server serving cert is signed for DNS names [localhost smithi059] and IPs [172.21.15.59 127.0.0.1 ::1] 2022-08-05T16:04:57.816 INFO:teuthology.orchestra.run.smithi059.stdout:[certs] Generating "etcd/peer" certificate and key 2022-08-05T16:04:57.820 INFO:teuthology.orchestra.run.smithi059.stdout:[certs] etcd/peer serving cert is signed for DNS names [localhost smithi059] and IPs [172.21.15.59 127.0.0.1 ::1] 2022-08-05T16:04:57.955 INFO:teuthology.orchestra.run.smithi059.stdout:[certs] Generating "etcd/healthcheck-client" certificate and key 2022-08-05T16:04:58.033 INFO:teuthology.orchestra.run.smithi059.stdout:[certs] Generating "apiserver-etcd-client" certificate and key 2022-08-05T16:04:58.111 INFO:teuthology.orchestra.run.smithi059.stdout:[certs] Generating "sa" key and public key 2022-08-05T16:04:58.112 INFO:teuthology.orchestra.run.smithi059.stdout:[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 2022-08-05T16:04:58.199 INFO:teuthology.orchestra.run.smithi059.stdout:[kubeconfig] Writing "admin.conf" kubeconfig file 2022-08-05T16:04:58.278 INFO:teuthology.orchestra.run.smithi059.stdout:[kubeconfig] Writing "kubelet.conf" kubeconfig file 2022-08-05T16:04:58.492 INFO:teuthology.orchestra.run.smithi059.stdout:[kubeconfig] Writing "controller-manager.conf" kubeconfig file 2022-08-05T16:04:58.697 INFO:teuthology.orchestra.run.smithi059.stdout:[kubeconfig] Writing "scheduler.conf" kubeconfig file 2022-08-05T16:04:58.767 INFO:teuthology.orchestra.run.smithi059.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-08-05T16:04:58.790 INFO:teuthology.orchestra.run.smithi059.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-08-05T16:04:58.790 INFO:teuthology.orchestra.run.smithi059.stdout:[kubelet-start] Starting the kubelet 2022-08-05T16:04:58.968 INFO:teuthology.orchestra.run.smithi059.stdout:[control-plane] Using manifest folder "/etc/kubernetes/manifests" 2022-08-05T16:04:58.968 INFO:teuthology.orchestra.run.smithi059.stdout:[control-plane] Creating static Pod manifest for "kube-apiserver" 2022-08-05T16:04:59.026 INFO:teuthology.orchestra.run.smithi059.stdout:[control-plane] Creating static Pod manifest for "kube-controller-manager" 2022-08-05T16:04:59.032 INFO:teuthology.orchestra.run.smithi059.stdout:[control-plane] Creating static Pod manifest for "kube-scheduler" 2022-08-05T16:04:59.033 INFO:teuthology.orchestra.run.smithi059.stdout:[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 2022-08-05T16:04:59.034 INFO:teuthology.orchestra.run.smithi059.stdout:[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 2022-08-05T16:05:17.979 INFO:teuthology.orchestra.run.smithi059.stdout:[apiclient] All control plane components are healthy after 19.003781 seconds 2022-08-05T16:05:17.980 INFO:teuthology.orchestra.run.smithi059.stdout:[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 2022-08-05T16:05:18.062 INFO:teuthology.orchestra.run.smithi059.stdout:[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster 2022-08-05T16:05:18.593 INFO:teuthology.orchestra.run.smithi059.stdout:[upload-certs] Skipping phase. Please see --upload-certs 2022-08-05T16:05:18.610 INFO:teuthology.orchestra.run.smithi059.stdout:[mark-control-plane] Marking the node smithi059 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] 2022-08-05T16:05:18.611 INFO:teuthology.orchestra.run.smithi059.stdout:[mark-control-plane] Marking the node smithi059 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule node-role.kubernetes.io/control-plane:NoSchedule] 2022-08-05T16:05:19.151 INFO:teuthology.orchestra.run.smithi059.stdout:[bootstrap-token] Using token: abcdef.6zedpdac7qr7of2m 2022-08-05T16:05:19.151 INFO:teuthology.orchestra.run.smithi059.stdout:[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 2022-08-05T16:05:19.266 INFO:teuthology.orchestra.run.smithi059.stdout:[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes 2022-08-05T16:05:19.268 INFO:teuthology.orchestra.run.smithi059.stdout:[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 2022-08-05T16:05:19.268 INFO:teuthology.orchestra.run.smithi059.stdout:[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 2022-08-05T16:05:19.272 INFO:teuthology.orchestra.run.smithi059.stdout:[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 2022-08-05T16:05:19.327 INFO:teuthology.orchestra.run.smithi059.stdout:[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 2022-08-05T16:05:19.388 INFO:teuthology.orchestra.run.smithi059.stdout:[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key 2022-08-05T16:05:19.783 INFO:teuthology.orchestra.run.smithi059.stdout:[addons] Applied essential addon: CoreDNS 2022-08-05T16:05:20.033 INFO:teuthology.orchestra.run.smithi059.stdout:[addons] Applied essential addon: kube-proxy 2022-08-05T16:05:20.033 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:05:20.045 INFO:teuthology.orchestra.run.smithi059.stdout:Your Kubernetes control-plane has initialized successfully! 2022-08-05T16:05:20.046 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:05:20.046 INFO:teuthology.orchestra.run.smithi059.stdout:To start using your cluster, you need to run the following as a regular user: 2022-08-05T16:05:20.046 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:05:20.047 INFO:teuthology.orchestra.run.smithi059.stdout: mkdir -p $HOME/.kube 2022-08-05T16:05:20.047 INFO:teuthology.orchestra.run.smithi059.stdout: sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 2022-08-05T16:05:20.047 INFO:teuthology.orchestra.run.smithi059.stdout: sudo chown $(id -u):$(id -g) $HOME/.kube/config 2022-08-05T16:05:20.048 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:05:20.050 INFO:teuthology.orchestra.run.smithi059.stdout:Alternatively, if you are the root user, you can run: 2022-08-05T16:05:20.050 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:05:20.051 INFO:teuthology.orchestra.run.smithi059.stdout: export KUBECONFIG=/etc/kubernetes/admin.conf 2022-08-05T16:05:20.051 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:05:20.052 INFO:teuthology.orchestra.run.smithi059.stdout:You should now deploy a pod network to the cluster. 2022-08-05T16:05:20.052 INFO:teuthology.orchestra.run.smithi059.stdout:Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 2022-08-05T16:05:20.053 INFO:teuthology.orchestra.run.smithi059.stdout: https://kubernetes.io/docs/concepts/cluster-administration/addons/ 2022-08-05T16:05:20.054 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:05:20.072 INFO:teuthology.orchestra.run.smithi059.stdout:Then you can join any number of worker nodes by running the following on each as root: 2022-08-05T16:05:20.072 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:05:20.072 INFO:teuthology.orchestra.run.smithi059.stdout:kubeadm join 172.21.15.59:6443 --token abcdef.6zedpdac7qr7of2m \ 2022-08-05T16:05:20.073 INFO:teuthology.orchestra.run.smithi059.stdout: --discovery-token-ca-cert-hash sha256:ee096fea60a7e665c25c1671e231aaa22503c836ab1ceea48609a35d383ba107 2022-08-05T16:05:20.103 DEBUG:teuthology.orchestra.run.smithi157:> sudo kubeadm join 172.21.15.59:6443 --node-name smithi157 --token abcdef.6zedpdac7qr7of2m --discovery-token-unsafe-skip-ca-verification 2022-08-05T16:05:20.191 DEBUG:teuthology.orchestra.run.smithi201:> sudo kubeadm join 172.21.15.59:6443 --node-name smithi201 --token abcdef.6zedpdac7qr7of2m --discovery-token-unsafe-skip-ca-verification 2022-08-05T16:05:20.241 INFO:teuthology.orchestra.run.smithi157.stdout:[preflight] Running pre-flight checks 2022-08-05T16:05:20.314 INFO:teuthology.orchestra.run.smithi201.stdout:[preflight] Running pre-flight checks 2022-08-05T16:05:32.706 INFO:teuthology.orchestra.run.smithi157.stdout:[preflight] Reading configuration from the cluster... 2022-08-05T16:05:32.707 INFO:teuthology.orchestra.run.smithi157.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-08-05T16:05:32.786 INFO:teuthology.orchestra.run.smithi157.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-08-05T16:05:32.864 INFO:teuthology.orchestra.run.smithi157.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-08-05T16:05:32.864 INFO:teuthology.orchestra.run.smithi157.stdout:[kubelet-start] Starting the kubelet 2022-08-05T16:05:33.063 INFO:teuthology.orchestra.run.smithi157.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-08-05T16:05:33.130 INFO:teuthology.orchestra.run.smithi201.stdout:[preflight] Reading configuration from the cluster... 2022-08-05T16:05:33.139 INFO:teuthology.orchestra.run.smithi201.stdout:[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-08-05T16:05:33.159 INFO:teuthology.orchestra.run.smithi201.stdout:[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" 2022-08-05T16:05:33.210 INFO:teuthology.orchestra.run.smithi201.stdout:[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" 2022-08-05T16:05:33.210 INFO:teuthology.orchestra.run.smithi201.stdout:[kubelet-start] Starting the kubelet 2022-08-05T16:05:33.420 INFO:teuthology.orchestra.run.smithi201.stdout:[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... 2022-08-05T16:05:38.586 INFO:teuthology.orchestra.run.smithi157.stdout: 2022-08-05T16:05:38.586 INFO:teuthology.orchestra.run.smithi157.stdout:This node has joined the cluster: 2022-08-05T16:05:38.587 INFO:teuthology.orchestra.run.smithi157.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-08-05T16:05:38.587 INFO:teuthology.orchestra.run.smithi157.stdout:* The Kubelet was informed of the new secure connection details. 2022-08-05T16:05:38.588 INFO:teuthology.orchestra.run.smithi157.stdout: 2022-08-05T16:05:38.613 INFO:teuthology.orchestra.run.smithi157.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-08-05T16:05:38.619 INFO:teuthology.orchestra.run.smithi157.stdout: 2022-08-05T16:05:46.505 INFO:teuthology.orchestra.run.smithi201.stdout: 2022-08-05T16:05:46.505 INFO:teuthology.orchestra.run.smithi201.stdout:This node has joined the cluster: 2022-08-05T16:05:46.506 INFO:teuthology.orchestra.run.smithi201.stdout:* Certificate signing request was sent to apiserver and a response was received. 2022-08-05T16:05:46.506 INFO:teuthology.orchestra.run.smithi201.stdout:* The Kubelet was informed of the new secure connection details. 2022-08-05T16:05:46.506 INFO:teuthology.orchestra.run.smithi201.stdout: 2022-08-05T16:05:46.507 INFO:teuthology.orchestra.run.smithi201.stdout:Run 'kubectl get nodes' on the control-plane to see this node join the cluster. 2022-08-05T16:05:46.507 INFO:teuthology.orchestra.run.smithi201.stdout: 2022-08-05T16:05:46.509 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-08-05T16:05:46.510 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/etc/kubernetes/admin.conf of=/dev/stdout 2022-08-05T16:05:46.599 INFO:tasks.kubeadm:Setting up kubectl 2022-08-05T16:05:46.600 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-08-05T16:05:46.664 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-08-05T16:05:46.796 DEBUG:teuthology.orchestra.run.smithi201:> mkdir -p .kube && sudo mkdir -p /root/.kube 2022-08-05T16:05:46.837 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-08-05T16:05:46.838 DEBUG:teuthology.orchestra.run.smithi059:> dd of=.kube/config 2022-08-05T16:05:46.928 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-08-05T16:05:46.987 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/root/.kube/config 2022-08-05T16:05:47.149 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2022-08-05T16:05:47.171 DEBUG:teuthology.orchestra.run.smithi157:> dd of=.kube/config 2022-08-05T16:05:47.312 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2022-08-05T16:05:47.312 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/root/.kube/config 2022-08-05T16:05:47.427 DEBUG:teuthology.orchestra.run.smithi201:> set -ex 2022-08-05T16:05:47.428 DEBUG:teuthology.orchestra.run.smithi201:> dd of=.kube/config 2022-08-05T16:05:47.506 DEBUG:teuthology.orchestra.run.smithi201:> set -ex 2022-08-05T16:05:47.506 DEBUG:teuthology.orchestra.run.smithi201:> sudo dd of=/root/.kube/config 2022-08-05T16:05:47.619 DEBUG:teuthology.orchestra.run.smithi059:> curl https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml 2022-08-05T16:05:47.783 INFO:teuthology.orchestra.run.smithi059.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2022-08-05T16:05:47.835 INFO:teuthology.orchestra.run.smithi059.stderr: Dload Upload Total Spent Left Speed 2022-08-05T16:05:47.879 INFO:teuthology.orchestra.run.smithi059.stdout:--- 2022-08-05T16:05:47.880 INFO:teuthology.orchestra.run.smithi059.stdout:kind: Namespace 2022-08-05T16:05:47.893 INFO:teuthology.orchestra.run.smithi059.stdout:apiVersion: v1 2022-08-05T16:05:47.906 INFO:teuthology.orchestra.run.smithi059.stdout:metadata: 2022-08-05T16:05:47.906 INFO:teuthology.orchestra.run.smithi059.stdout: name: kube-flannel 2022-08-05T16:05:47.907 INFO:teuthology.orchestra.run.smithi059.stdout: labels: 2022-08-05T16:05:47.907 INFO:teuthology.orchestra.run.smithi059.stdout: pod-security.kubernetes.io/enforce: privileged 2022-08-05T16:05:47.908 INFO:teuthology.orchestra.run.smithi059.stdout:--- 2022-08-05T16:05:47.908 INFO:teuthology.orchestra.run.smithi059.stdout:kind: ClusterRole 2022-08-05T16:05:47.909 INFO:teuthology.orchestra.run.smithi059.stdout:apiVersion: rbac.authorization.k8s.io/v1 2022-08-05T16:05:47.909 INFO:teuthology.orchestra.run.smithi059.stdout:metadata: 2022-08-05T16:05:47.964 INFO:teuthology.orchestra.run.smithi059.stdout: name: flannel 2022-08-05T16:05:47.970 INFO:teuthology.orchestra.run.smithi059.stdout:rules: 2022-08-05T16:05:47.971 INFO:teuthology.orchestra.run.smithi059.stdout:- apiGroups: 2022-08-05T16:05:47.971 INFO:teuthology.orchestra.run.smithi059.stdout: - "" 2022-08-05T16:05:47.972 INFO:teuthology.orchestra.run.smithi059.stdout: resources: 2022-08-05T16:05:47.972 INFO:teuthology.orchestra.run.smithi059.stdout: - pods 2022-08-05T16:05:47.973 INFO:teuthology.orchestra.run.smithi059.stdout: verbs: 2022-08-05T16:05:47.973 INFO:teuthology.orchestra.run.smithi059.stdout: - get 2022-08-05T16:05:47.977 INFO:teuthology.orchestra.run.smithi059.stdout:- apiGroups: 2022-08-05T16:05:47.977 INFO:teuthology.orchestra.run.smithi059.stdout: - "" 2022-08-05T16:05:47.978 INFO:teuthology.orchestra.run.smithi059.stdout: resources: 2022-08-05T16:05:47.978 INFO:teuthology.orchestra.run.smithi059.stdout: - nodes 2022-08-05T16:05:47.979 INFO:teuthology.orchestra.run.smithi059.stdout: verbs: 2022-08-05T16:05:47.979 INFO:teuthology.orchestra.run.smithi059.stdout: - list 2022-08-05T16:05:47.979 INFO:teuthology.orchestra.run.smithi059.stdout: - watch 2022-08-05T16:05:47.980 INFO:teuthology.orchestra.run.smithi059.stdout:- apiGroups: 2022-08-05T16:05:47.980 INFO:teuthology.orchestra.run.smithi059.stdout: - "" 2022-08-05T16:05:47.981 INFO:teuthology.orchestra.run.smithi059.stdout: resources: 2022-08-05T16:05:47.982 INFO:teuthology.orchestra.run.smithi059.stdout: - nodes/status 2022-08-05T16:05:48.037 INFO:teuthology.orchestra.run.smithi059.stdout: verbs: 2022-08-05T16:05:48.037 INFO:teuthology.orchestra.run.smithi059.stdout: - patch 2022-08-05T16:05:48.039 INFO:teuthology.orchestra.run.smithi059.stdout:--- 2022-08-05T16:05:48.040 INFO:teuthology.orchestra.run.smithi059.stdout:kind: ClusterRoleBinding 2022-08-05T16:05:48.040 INFO:teuthology.orchestra.run.smithi059.stdout:apiVersion: rbac.authorization.k8s.io/v1 2022-08-05T16:05:48.041 INFO:teuthology.orchestra.run.smithi059.stdout:metadata: 2022-08-05T16:05:48.042 INFO:teuthology.orchestra.run.smithi059.stdout: name: flannel 2022-08-05T16:05:48.042 INFO:teuthology.orchestra.run.smithi059.stdout:roleRef: 2022-08-05T16:05:48.042 INFO:teuthology.orchestra.run.smithi059.stdout: apiGroup: rbac.authorization.k8s.io 2022-08-05T16:05:48.057 INFO:teuthology.orchestra.run.smithi059.stdout: kind: ClusterRole 2022-08-05T16:05:48.065 INFO:teuthology.orchestra.run.smithi059.stdout: name: flannel 2022-08-05T16:05:48.065 INFO:teuthology.orchestra.run.smithi059.stdout:subjects: 2022-08-05T16:05:48.066 INFO:teuthology.orchestra.run.smithi059.stdout:- kind: ServiceAccount 2022-08-05T16:05:48.066 INFO:teuthology.orchestra.run.smithi059.stdout: name: flannel 2022-08-05T16:05:48.066 INFO:teuthology.orchestra.run.smithi059.stdout: namespace: kube-flannel 2022-08-05T16:05:48.067 INFO:teuthology.orchestra.run.smithi059.stdout:--- 2022-08-05T16:05:48.067 INFO:teuthology.orchestra.run.smithi059.stdout:apiVersion: v1 2022-08-05T16:05:48.067 INFO:teuthology.orchestra.run.smithi059.stdout:kind: ServiceAccount 2022-08-05T16:05:48.069 INFO:teuthology.orchestra.run.smithi059.stdout:metadata: 2022-08-05T16:05:48.070 INFO:teuthology.orchestra.run.smithi059.stdout: name: flannel 2022-08-05T16:05:48.071 INFO:teuthology.orchestra.run.smithi059.stdout: namespace: kube-flannel 2022-08-05T16:05:48.076 INFO:teuthology.orchestra.run.smithi059.stdout:--- 2022-08-05T16:05:48.086 INFO:teuthology.orchestra.run.smithi059.stdout:kind: ConfigMap 2022-08-05T16:05:48.086 INFO:teuthology.orchestra.run.smithi059.stdout:apiVersion: v1 2022-08-05T16:05:48.087 INFO:teuthology.orchestra.run.smithi059.stdout:metadata: 2022-08-05T16:05:48.087 INFO:teuthology.orchestra.run.smithi059.stdout: name: kube-flannel-cfg 2022-08-05T16:05:48.087 INFO:teuthology.orchestra.run.smithi059.stdout: namespace: kube-flannel 2022-08-05T16:05:48.088 INFO:teuthology.orchestra.run.smithi059.stdout: labels: 2022-08-05T16:05:48.088 INFO:teuthology.orchestra.run.smithi059.stdout: tier: node 2022-08-05T16:05:48.088 INFO:teuthology.orchestra.run.smithi059.stdout: app: flannel 2022-08-05T16:05:48.088 INFO:teuthology.orchestra.run.smithi059.stdout:data: 2022-08-05T16:05:48.089 INFO:teuthology.orchestra.run.smithi059.stdout: cni-conf.json: | 2022-08-05T16:05:48.089 INFO:teuthology.orchestra.run.smithi059.stdout: { 2022-08-05T16:05:48.089 INFO:teuthology.orchestra.run.smithi059.stdout: "name": "cbr0", 2022-08-05T16:05:48.091 INFO:teuthology.orchestra.run.smithi059.stdout: "cniVersion": "0.3.1", 2022-08-05T16:05:48.099 INFO:teuthology.orchestra.run.smithi059.stdout: "plugins": [ 2022-08-05T16:05:48.100 INFO:teuthology.orchestra.run.smithi059.stdout: { 2022-08-05T16:05:48.100 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "flannel", 2022-08-05T16:05:48.132 INFO:teuthology.orchestra.run.smithi059.stdout: "delegate": { 2022-08-05T16:05:48.132 INFO:teuthology.orchestra.run.smithi059.stdout: "hairpinMode": true, 2022-08-05T16:05:48.132 INFO:teuthology.orchestra.run.smithi059.stdout: "isDefaultGateway": true 2022-08-05T16:05:48.133 INFO:teuthology.orchestra.run.smithi059.stdout: } 2022-08-05T16:05:48.133 INFO:teuthology.orchestra.run.smithi059.stdout: }, 2022-08-05T16:05:48.133 INFO:teuthology.orchestra.run.smithi059.stdout: { 2022-08-05T16:05:48.151 INFO:teuthology.orchestra.run.smithi059.stdout: "type": "portmap", 2022-08-05T16:05:48.151 INFO:teuthology.orchestra.run.smithi059.stdout: "capabilities": { 2022-08-05T16:05:48.152 INFO:teuthology.orchestra.run.smithi059.stdout: "portMappings": true 2022-08-05T16:05:48.152 INFO:teuthology.orchestra.run.smithi059.stdout: } 2022-08-05T16:05:48.153 INFO:teuthology.orchestra.run.smithi059.stdout: } 2022-08-05T16:05:48.153 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2022-08-05T16:05:48.162 INFO:teuthology.orchestra.run.smithi059.stdout: } 2022-08-05T16:05:48.197 INFO:teuthology.orchestra.run.smithi059.stdout: net-conf.json: | 2022-08-05T16:05:48.198 INFO:teuthology.orchestra.run.smithi059.stdout: { 2022-08-05T16:05:48.198 INFO:teuthology.orchestra.run.smithi059.stdout: "Network": "10.244.0.0/16", 2022-08-05T16:05:48.199 INFO:teuthology.orchestra.run.smithi059.stdout: "Backend": { 2022-08-05T16:05:48.199 INFO:teuthology.orchestra.run.smithi059.stdout: "Type": "vxlan" 2022-08-05T16:05:48.200 INFO:teuthology.orchestra.run.smithi059.stdout: } 2022-08-05T16:05:48.200 INFO:teuthology.orchestra.run.smithi059.stdout: } 2022-08-05T16:05:48.201 INFO:teuthology.orchestra.run.smithi059.stdout:--- 2022-08-05T16:05:48.201 INFO:teuthology.orchestra.run.smithi059.stdout:apiVersion: apps/v1 2022-08-05T16:05:48.202 INFO:teuthology.orchestra.run.smithi059.stdout:kind: DaemonSet 2022-08-05T16:05:48.202 INFO:teuthology.orchestra.run.smithi059.stdout:metadata: 2022-08-05T16:05:48.203 INFO:teuthology.orchestra.run.smithi059.stdout: name: kube-flannel-ds 2022-08-05T16:05:48.203 INFO:teuthology.orchestra.run.smithi059.stdout: namespace: kube-flannel 2022-08-05T16:05:48.204 INFO:teuthology.orchestra.run.smithi059.stdout: labels: 2022-08-05T16:05:48.204 INFO:teuthology.orchestra.run.smithi059.stdout: tier: node 2022-08-05T16:05:48.205 INFO:teuthology.orchestra.run.smithi059.stdout: app: flannel 2022-08-05T16:05:48.205 INFO:teuthology.orchestra.run.smithi059.stdout:spec: 2022-08-05T16:05:48.206 INFO:teuthology.orchestra.run.smithi059.stdout: selector: 2022-08-05T16:05:48.206 INFO:teuthology.orchestra.run.smithi059.stdout: matchLabels: 2022-08-05T16:05:48.206 INFO:teuthology.orchestra.run.smithi059.stdout: app: flannel 2022-08-05T16:05:48.207 INFO:teuthology.orchestra.run.smithi059.stdout: template: 2022-08-05T16:05:48.207 INFO:teuthology.orchestra.run.smithi059.stdout: metadata: 2022-08-05T16:05:48.208 INFO:teuthology.orchestra.run.smithi059.stdout: labels: 2022-08-05T16:05:48.208 INFO:teuthology.orchestra.run.smithi059.stdout: tier: node 2022-08-05T16:05:48.209 INFO:teuthology.orchestra.run.smithi059.stdout: app: flannel 2022-08-05T16:05:48.209 INFO:teuthology.orchestra.run.smithi059.stdout: spec: 2022-08-05T16:05:48.210 INFO:teuthology.orchestra.run.smithi059.stdout: affinity: 2022-08-05T16:05:48.210 INFO:teuthology.orchestra.run.smithi059.stdout: nodeAffinity: 2022-08-05T16:05:48.211 INFO:teuthology.orchestra.run.smithi059.stdout: requiredDuringSchedulingIgnoredDuringExecution: 2022-08-05T16:05:48.212 INFO:teuthology.orchestra.run.smithi059.stdout: nodeSelectorTerms: 2022-08-05T16:05:48.212 INFO:teuthology.orchestra.run.smithi059.stdout: - matchExpressions: 2022-08-05T16:05:48.213 INFO:teuthology.orchestra.run.smithi059.stdout: - key: kubernetes.io/os 2022-08-05T16:05:48.213 INFO:teuthology.orchestra.run.smithi059.stdout: operator: In 2022-08-05T16:05:48.214 INFO:teuthology.orchestra.run.smithi059.stdout: values: 2022-08-05T16:05:48.214 INFO:teuthology.orchestra.run.smithi059.stdout: - linux 2022-08-05T16:05:48.215 INFO:teuthology.orchestra.run.smithi059.stdout: hostNetwork: true 2022-08-05T16:05:48.215 INFO:teuthology.orchestra.run.smithi059.stdout: priorityClassName: system-node-critical 2022-08-05T16:05:48.215 INFO:teuthology.orchestra.run.smithi059.stdout: tolerations: 2022-08-05T16:05:48.216 INFO:teuthology.orchestra.run.smithi059.stdout: - operator: Exists 2022-08-05T16:05:48.216 INFO:teuthology.orchestra.run.smithi059.stdout: effect: NoSchedule 2022-08-05T16:05:48.217 INFO:teuthology.orchestra.run.smithi059.stdout: serviceAccountName: flannel 2022-08-05T16:05:48.219 INFO:teuthology.orchestra.run.smithi059.stdout: initContainers: 2022-08-05T16:05:48.220 INFO:teuthology.orchestra.run.smithi059.stdout: - name: install-cni-plugin 2022-08-05T16:05:48.221 INFO:teuthology.orchestra.run.smithi059.stdout: #image: flannelcni/flannel-cni-plugin:v1.1.0 for ppc64le and mips64le (dockerhub limitations may apply) 2022-08-05T16:05:48.221 INFO:teuthology.orchestra.run.smithi059.stdout: image: docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0 2022-08-05T16:05:48.222 INFO:teuthology.orchestra.run.smithi059.stdout: command: 2022-08-05T16:05:48.235 INFO:teuthology.orchestra.run.smithi059.stdout: - cp 2022-08-05T16:05:48.235 INFO:teuthology.orchestra.run.smithi059.stdout: args: 2022-08-05T16:05:48.236 INFO:teuthology.orchestra.run.smithi059.stdout: - -f 2022-08-05T16:05:48.236 INFO:teuthology.orchestra.run.smithi059.stdout: - /flannel 2022-08-05T16:05:48.237 INFO:teuthology.orchestra.run.smithi059.stdout: - /opt/cni/bin/flannel 2022-08-05T16:05:48.237 INFO:teuthology.orchestra.run.smithi059.stdout: volumeMounts: 2022-08-05T16:05:48.238 INFO:teuthology.orchestra.run.smithi059.stdout: - name: cni-plugin 2022-08-05T16:05:48.239 INFO:teuthology.orchestra.run.smithi059.stdout: mountPath: /opt/cni/bin 2022-08-05T16:05:48.239 INFO:teuthology.orchestra.run.smithi059.stdout: - name: install-cni 2022-08-05T16:05:48.239 INFO:teuthology.orchestra.run.smithi059.stdout: #image: flannelcni/flannel:v0.19.0 for ppc64le and mips64le (dockerhub limitations may apply) 2022-08-05T16:05:48.240 INFO:teuthology.orchestra.run.smithi059.stdout: image: docker.io/rancher/mirrored-flannelcni-flannel:v0.19.0 2022-08-05T16:05:48.240 INFO:teuthology.orchestra.run.smithi059.stdout: command: 2022-08-05T16:05:48.241 INFO:teuthology.orchestra.run.smithi059.stdout: - cp 2022-08-05T16:05:48.241 INFO:teuthology.orchestra.run.smithi059.stdout: args: 2022-08-05T16:05:48.242 INFO:teuthology.orchestra.run.smithi059.stdout: - -f 2022-08-05T16:05:48.248 INFO:teuthology.orchestra.run.smithi059.stdout: - /etc/kube-flannel/cni-conf.json 2022-08-05T16:05:48.249 INFO:teuthology.orchestra.run.smithi059.stdout: - /etc/cni/net.d/10-flannel.conflist 2022-08-05T16:05:48.249 INFO:teuthology.orchestra.run.smithi059.stdout: volumeMounts: 2022-08-05T16:05:48.249 INFO:teuthology.orchestra.run.smithi059.stdout: - name: cni 2022-08-05T16:05:48.253 INFO:teuthology.orchestra.run.smithi059.stdout: mountPath: /etc/cni/net.d 2022-08-05T16:05:48.254 INFO:teuthology.orchestra.run.smithi059.stdout: - name: flannel-cfg 2022-08-05T16:05:48.254 INFO:teuthology.orchestra.run.smithi059.stdout: mountPath: /etc/kube-flannel/ 2022-08-05T16:05:48.255 INFO:teuthology.orchestra.run.smithi059.stdout: containers: 2022-08-05T16:05:48.255 INFO:teuthology.orchestra.run.smithi059.stdout: - name: kube-flannel 2022-08-05T16:05:48.256 INFO:teuthology.orchestra.run.smithi059.stdout: #image: flannelcni/flannel:v0.19.0 for ppc64le and mips64le (dockerhub limitations may apply) 2022-08-05T16:05:48.256 INFO:teuthology.orchestra.run.smithi059.stdout: image: docker.io/rancher/mirrored-flannelcni-flannel:v0.19.0 2022-08-05T16:05:48.256 INFO:teuthology.orchestra.run.smithi059.stdout: command: 2022-08-05T16:05:48.257 INFO:teuthology.orchestra.run.smithi059.stdout: - /opt/bin/flanneld 2022-08-05T16:05:48.257 INFO:teuthology.orchestra.run.smithi059.stdout: args: 2022-08-05T16:05:48.257 INFO:teuthology.orchestra.run.smithi059.stdout: - --ip-masq 2022-08-05T16:05:48.258 INFO:teuthology.orchestra.run.smithi059.stdout: - --kube-subnet-mgr 2022-08-05T16:05:48.258 INFO:teuthology.orchestra.run.smithi059.stdout: resources: 2022-08-05T16:05:48.259 INFO:teuthology.orchestra.run.smithi059.stdout: requests: 2022-08-05T16:05:48.259 INFO:teuthology.orchestra.run.smithi059.stdout: cpu: "100m" 2022-08-05T16:05:48.260 INFO:teuthology.orchestra.run.smithi059.stdout: memory: "50Mi" 2022-08-05T16:05:48.260 INFO:teuthology.orchestra.run.smithi059.stdout: limits: 2022-08-05T16:05:48.260 INFO:teuthology.orchestra.run.smithi059.stdout: cpu: "100m" 2022-08-05T16:05:48.261 INFO:teuthology.orchestra.run.smithi059.stdout: memory: "50Mi" 2022-08-05T16:05:48.261 INFO:teuthology.orchestra.run.smithi059.stdout: securityContext: 2022-08-05T16:05:48.284 INFO:teuthology.orchestra.run.smithi059.stdout: privileged: false 2022-08-05T16:05:48.293 INFO:teuthology.orchestra.run.smithi059.stdout: capabilities: 2022-08-05T16:05:48.293 INFO:teuthology.orchestra.run.smithi059.stdout: add: ["NET_ADMIN", "NET_RAW"] 2022-08-05T16:05:48.294 INFO:teuthology.orchestra.run.smithi059.stdout: env: 2022-08-05T16:05:48.294 INFO:teuthology.orchestra.run.smithi059.stdout: - name: POD_NAME 2022-08-05T16:05:48.295 INFO:teuthology.orchestra.run.smithi059.stdout: valueFrom: 2022-08-05T16:05:48.295 INFO:teuthology.orchestra.run.smithi059.stdout: fieldRef: 2022-08-05T16:05:48.296 INFO:teuthology.orchestra.run.smithi059.stdout: fieldPath: metadata.name 2022-08-05T16:05:48.296 INFO:teuthology.orchestra.run.smithi059.stdout: - name: POD_NAMESPACE 2022-08-05T16:05:48.296 INFO:teuthology.orchestra.run.smithi059.stdout: valueFrom: 2022-08-05T16:05:48.312 INFO:teuthology.orchestra.run.smithi059.stdout: fieldRef: 2022-08-05T16:05:48.312 INFO:teuthology.orchestra.run.smithi059.stdout: fieldPath: metadata.namespace 2022-08-05T16:05:48.312 INFO:teuthology.orchestra.run.smithi059.stdout: - name: EVENT_QUEUE_DEPTH 2022-08-05T16:05:48.313 INFO:teuthology.orchestra.run.smithi059.stdout: value: "5000" 2022-08-05T16:05:48.313 INFO:teuthology.orchestra.run.smithi059.stdout: volumeMounts: 2022-08-05T16:05:48.313 INFO:teuthology.orchestra.run.smithi059.stdout: - name: run 2022-08-05T16:05:48.313 INFO:teuthology.orchestra.run.smithi059.stdout: mountPath: /run/flannel 2022-08-05T16:05:48.315 INFO:teuthology.orchestra.run.smithi059.stdout: - name: flannel-cfg 2022-08-05T16:05:48.326 INFO:teuthology.orchestra.run.smithi059.stdout: mountPath: /etc/kube-flannel/ 2022-08-05T16:05:48.328 INFO:teuthology.orchestra.run.smithi059.stdout: - name: xtables-lock 2022-08-05T16:05:48.329 INFO:teuthology.orchestra.run.smithi059.stdout: mountPath: /run/xtables.lock 2022-08-05T16:05:48.329 INFO:teuthology.orchestra.run.smithi059.stdout: volumes: 2022-08-05T16:05:48.329 INFO:teuthology.orchestra.run.smithi059.stdout: - name: run 2022-08-05T16:05:48.329 INFO:teuthology.orchestra.run.smithi059.stdout: hostPath: 2022-08-05T16:05:48.330 INFO:teuthology.orchestra.run.smithi059.stdout: path: /run/flannel 2022-08-05T16:05:48.330 INFO:teuthology.orchestra.run.smithi059.stdout: - name: cni-plugin 2022-08-05T16:05:48.330 INFO:teuthology.orchestra.run.smithi059.stdout: hostPath: 2022-08-05T16:05:48.330 INFO:teuthology.orchestra.run.smithi059.stdout: path: /opt/cni/bin 2022-08-05T16:05:48.335 INFO:teuthology.orchestra.run.smithi059.stdout: - name: cni 2022-08-05T16:05:48.339 INFO:teuthology.orchestra.run.smithi059.stdout: hostPath: 2022-08-05T16:05:48.340 INFO:teuthology.orchestra.run.smithi059.stdout: path: /etc/cni/net.d 2022-08-05T16:05:48.340 INFO:teuthology.orchestra.run.smithi059.stdout: - name: flannel-cfg 2022-08-05T16:05:48.341 INFO:teuthology.orchestra.run.smithi059.stdout: configMap: 2022-08-05T16:05:48.365 INFO:teuthology.orchestra.run.smithi059.stdout: name: kube-flannel-cfg 2022-08-05T16:05:48.374 INFO:teuthology.orchestra.run.smithi059.stdout: - name: xtables-lock 2022-08-05T16:05:48.423 INFO:teuthology.orchestra.run.smithi059.stdout: hostPath: 2022-08-05T16:05:48.423 INFO:teuthology.orchestra.run.smithi059.stdout: path: /run/xtables.lock 2022-08-05T16:05:48.423 INFO:teuthology.orchestra.run.smithi059.stdout: type: FileOrCreate 2022-08-05T16:05:48.443 INFO:teuthology.orchestra.run.smithi059.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 4583 100 4583 0 0 40919 0 --:--:-- --:--:-- --:--:-- 40919 2022-08-05T16:05:48.659 INFO:tasks.kubeadm:Updating {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'kube-flannel-cfg', 'namespace': 'kube-flannel', 'labels': {'tier': 'node', 'app': 'flannel'}}, 'data': {'cni-conf.json': '{\n "name": "cbr0",\n "cniVersion": "0.3.1",\n "plugins": [\n {\n "type": "flannel",\n "delegate": {\n "hairpinMode": true,\n "isDefaultGateway": true\n }\n },\n {\n "type": "portmap",\n "capabilities": {\n "portMappings": true\n }\n }\n ]\n}\n', 'net-conf.json': '{\n "Network": "10.244.0.0/16",\n "Backend": {\n "Type": "vxlan"\n }\n}\n'}} 2022-08-05T16:05:48.660 INFO:tasks.kubeadm:Now {'kind': 'ConfigMap', 'apiVersion': 'v1', 'metadata': {'name': 'kube-flannel-cfg', 'namespace': 'kube-flannel', 'labels': {'tier': 'node', 'app': 'flannel'}}, 'data': {'cni-conf.json': '{\n "name": "cbr0",\n "cniVersion": "0.3.1",\n "plugins": [\n {\n "type": "flannel",\n "delegate": {\n "hairpinMode": true,\n "isDefaultGateway": true\n }\n },\n {\n "type": "portmap",\n "capabilities": {\n "portMappings": true\n }\n }\n ]\n}\n', 'net-conf.json': '{\n "Network": "10.249.208.0/21",\n "Backend": {\n "Type": "vxlan"\n }\n}\n'}} 2022-08-05T16:05:48.901 DEBUG:tasks.kubeadm:Flannel: apiVersion: v1 kind: Namespace metadata: labels: pod-security.kubernetes.io/enforce: privileged name: kube-flannel --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: flannel rules: - apiGroups: - '' resources: - pods verbs: - get - apiGroups: - '' resources: - nodes verbs: - list - watch - apiGroups: - '' resources: - nodes/status verbs: - patch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: flannel roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannel subjects: - kind: ServiceAccount name: flannel namespace: kube-flannel --- apiVersion: v1 kind: ServiceAccount metadata: name: flannel namespace: kube-flannel --- apiVersion: v1 data: cni-conf.json: "{\n \"name\": \"cbr0\",\n \"cniVersion\": \"0.3.1\",\n \"plugins\"\ : [\n {\n \"type\": \"flannel\",\n \"delegate\": {\n \"hairpinMode\"\ : true,\n \"isDefaultGateway\": true\n }\n },\n {\n \"\ type\": \"portmap\",\n \"capabilities\": {\n \"portMappings\": true\n\ \ }\n }\n ]\n}\n" net-conf.json: "{\n \"Network\": \"10.249.208.0/21\",\n \"Backend\": {\n \"\ Type\": \"vxlan\"\n }\n}\n" kind: ConfigMap metadata: labels: app: flannel tier: node name: kube-flannel-cfg namespace: kube-flannel --- apiVersion: apps/v1 kind: DaemonSet metadata: labels: app: flannel tier: node name: kube-flannel-ds namespace: kube-flannel spec: selector: matchLabels: app: flannel template: metadata: labels: app: flannel tier: node spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - linux containers: - args: - --ip-masq - --kube-subnet-mgr command: - /opt/bin/flanneld env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: EVENT_QUEUE_DEPTH value: '5000' image: docker.io/rancher/mirrored-flannelcni-flannel:v0.19.0 name: kube-flannel resources: limits: cpu: 100m memory: 50Mi requests: cpu: 100m memory: 50Mi securityContext: capabilities: add: - NET_ADMIN - NET_RAW privileged: false volumeMounts: - mountPath: /run/flannel name: run - mountPath: /etc/kube-flannel/ name: flannel-cfg - mountPath: /run/xtables.lock name: xtables-lock hostNetwork: true initContainers: - args: - -f - /flannel - /opt/cni/bin/flannel command: - cp image: docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0 name: install-cni-plugin volumeMounts: - mountPath: /opt/cni/bin name: cni-plugin - args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist command: - cp image: docker.io/rancher/mirrored-flannelcni-flannel:v0.19.0 name: install-cni volumeMounts: - mountPath: /etc/cni/net.d name: cni - mountPath: /etc/kube-flannel/ name: flannel-cfg priorityClassName: system-node-critical serviceAccountName: flannel tolerations: - effect: NoSchedule operator: Exists volumes: - hostPath: path: /run/flannel name: run - hostPath: path: /opt/cni/bin name: cni-plugin - hostPath: path: /etc/cni/net.d name: cni - configMap: name: kube-flannel-cfg name: flannel-cfg - hostPath: path: /run/xtables.lock type: FileOrCreate name: xtables-lock 2022-08-05T16:05:48.923 DEBUG:teuthology.orchestra.run.smithi059:> kubectl apply -f - 2022-08-05T16:05:51.408 INFO:teuthology.orchestra.run.smithi059.stdout:namespace/kube-flannel created 2022-08-05T16:05:51.453 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrole.rbac.authorization.k8s.io/flannel created 2022-08-05T16:05:51.457 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrolebinding.rbac.authorization.k8s.io/flannel created 2022-08-05T16:05:51.488 INFO:teuthology.orchestra.run.smithi059.stdout:serviceaccount/flannel created 2022-08-05T16:05:51.540 INFO:teuthology.orchestra.run.smithi059.stdout:configmap/kube-flannel-cfg created 2022-08-05T16:05:51.579 INFO:teuthology.orchestra.run.smithi059.stdout:daemonset.apps/kube-flannel-ds created 2022-08-05T16:05:51.591 INFO:tasks.kubeadm:Scanning for scratch devices 2022-08-05T16:05:51.592 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-08-05T16:05:51.592 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/scratch_devs of=/dev/stdout 2022-08-05T16:05:51.624 INFO:tasks.kubeadm:Scratch devices on smithi059: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-08-05T16:05:51.625 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-08-05T16:05:51.812 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records in 2022-08-05T16:05:51.812 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records out 2022-08-05T16:05:51.835 INFO:teuthology.orchestra.run.smithi059.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0138124 s, 759 MB/s 2022-08-05T16:05:51.837 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-08-05T16:05:51.932 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records in 2022-08-05T16:05:51.933 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records out 2022-08-05T16:05:51.933 INFO:teuthology.orchestra.run.smithi059.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0171564 s, 611 MB/s 2022-08-05T16:05:51.973 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-08-05T16:05:52.055 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records in 2022-08-05T16:05:52.055 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records out 2022-08-05T16:05:52.056 INFO:teuthology.orchestra.run.smithi059.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0163922 s, 640 MB/s 2022-08-05T16:05:52.057 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-08-05T16:05:52.151 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records in 2022-08-05T16:05:52.151 INFO:teuthology.orchestra.run.smithi059.stderr:10+0 records out 2022-08-05T16:05:52.152 INFO:teuthology.orchestra.run.smithi059.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0164228 s, 638 MB/s 2022-08-05T16:05:52.184 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2022-08-05T16:05:52.184 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/scratch_devs of=/dev/stdout 2022-08-05T16:05:52.220 INFO:tasks.kubeadm:Scratch devices on smithi157: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-08-05T16:05:52.221 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-08-05T16:05:52.265 INFO:teuthology.orchestra.run.smithi157.stderr:10+0 records in 2022-08-05T16:05:52.266 INFO:teuthology.orchestra.run.smithi157.stderr:10+0 records out 2022-08-05T16:05:52.266 INFO:teuthology.orchestra.run.smithi157.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0147291 s, 712 MB/s 2022-08-05T16:05:52.267 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-08-05T16:05:52.379 INFO:teuthology.orchestra.run.smithi157.stderr:10+0 records in 2022-08-05T16:05:52.380 INFO:teuthology.orchestra.run.smithi157.stderr:10+0 records out 2022-08-05T16:05:52.380 INFO:teuthology.orchestra.run.smithi157.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0109882 s, 954 MB/s 2022-08-05T16:05:52.381 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-08-05T16:05:52.453 INFO:teuthology.orchestra.run.smithi157.stderr:10+0 records in 2022-08-05T16:05:52.464 INFO:teuthology.orchestra.run.smithi157.stderr:10+0 records out 2022-08-05T16:05:52.465 INFO:teuthology.orchestra.run.smithi157.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0148324 s, 707 MB/s 2022-08-05T16:05:52.465 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-08-05T16:05:52.586 INFO:teuthology.orchestra.run.smithi157.stderr:10+0 records in 2022-08-05T16:05:52.587 INFO:teuthology.orchestra.run.smithi157.stderr:10+0 records out 2022-08-05T16:05:52.587 INFO:teuthology.orchestra.run.smithi157.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0138363 s, 758 MB/s 2022-08-05T16:05:52.602 DEBUG:teuthology.orchestra.run.smithi201:> set -ex 2022-08-05T16:05:52.603 DEBUG:teuthology.orchestra.run.smithi201:> dd if=/scratch_devs of=/dev/stdout 2022-08-05T16:05:52.628 INFO:tasks.kubeadm:Scratch devices on smithi201: ['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-08-05T16:05:52.628 DEBUG:teuthology.orchestra.run.smithi201:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 2022-08-05T16:05:52.707 INFO:teuthology.orchestra.run.smithi201.stderr:10+0 records in 2022-08-05T16:05:52.709 INFO:teuthology.orchestra.run.smithi201.stderr:10+0 records out 2022-08-05T16:05:52.710 INFO:teuthology.orchestra.run.smithi201.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.017982 s, 583 MB/s 2022-08-05T16:05:52.712 DEBUG:teuthology.orchestra.run.smithi201:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 2022-08-05T16:05:52.782 INFO:teuthology.orchestra.run.smithi201.stderr:10+0 records in 2022-08-05T16:05:52.812 INFO:teuthology.orchestra.run.smithi201.stderr:10+0 records out 2022-08-05T16:05:52.813 INFO:teuthology.orchestra.run.smithi201.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0159971 s, 655 MB/s 2022-08-05T16:05:52.874 DEBUG:teuthology.orchestra.run.smithi201:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 2022-08-05T16:05:52.943 INFO:teuthology.orchestra.run.smithi201.stderr:10+0 records in 2022-08-05T16:05:52.944 INFO:teuthology.orchestra.run.smithi201.stderr:10+0 records out 2022-08-05T16:05:52.950 INFO:teuthology.orchestra.run.smithi201.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0150819 s, 695 MB/s 2022-08-05T16:05:52.951 DEBUG:teuthology.orchestra.run.smithi201:> sudo dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 2022-08-05T16:05:53.092 INFO:teuthology.orchestra.run.smithi201.stderr:10+0 records in 2022-08-05T16:05:53.177 INFO:teuthology.orchestra.run.smithi201.stderr:10+0 records out 2022-08-05T16:05:53.177 INFO:teuthology.orchestra.run.smithi201.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0177486 s, 591 MB/s 2022-08-05T16:05:53.937 INFO:tasks.kubeadm:Creating PVs + StorageClass 2022-08-05T16:05:53.995 DEBUG:tasks.kubeadm:apiVersion: v1 kind: PersistentVolume metadata: name: smithi059-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi059 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi059-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi059 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi059-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi059 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi059-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi059 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi157-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi157 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi157-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi157 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi157-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi157 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi157-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi157 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi201-lv-1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_1 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi201 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi201-lv-2 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_2 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi201 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi201-lv-3 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_3 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi201 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: v1 kind: PersistentVolume metadata: name: smithi201-lv-4 spec: accessModes: - ReadWriteOnce capacity: storage: 100Gi local: path: /dev/vg_nvme/lv_4 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - smithi201 persistentVolumeReclaimPolicy: Retain storageClassName: scratch volumeMode: Block --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: scratch provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer 2022-08-05T16:05:54.045 DEBUG:teuthology.orchestra.run.smithi059:> kubectl create -f - 2022-08-05T16:05:54.423 INFO:teuthology.orchestra.run.smithi059.stdout:persistentvolume/smithi059-lv-1 created 2022-08-05T16:05:54.457 INFO:teuthology.orchestra.run.smithi059.stdout:persistentvolume/smithi059-lv-2 created 2022-08-05T16:05:54.511 INFO:teuthology.orchestra.run.smithi059.stdout:persistentvolume/smithi059-lv-3 created 2022-08-05T16:05:54.512 INFO:teuthology.orchestra.run.smithi059.stdout:persistentvolume/smithi059-lv-4 created 2022-08-05T16:05:54.543 INFO:teuthology.orchestra.run.smithi059.stdout:persistentvolume/smithi157-lv-1 created 2022-08-05T16:05:54.543 INFO:teuthology.orchestra.run.smithi059.stdout:persistentvolume/smithi157-lv-2 created 2022-08-05T16:05:54.599 INFO:teuthology.orchestra.run.smithi059.stdout:persistentvolume/smithi157-lv-3 created 2022-08-05T16:05:54.616 INFO:teuthology.orchestra.run.smithi059.stdout:persistentvolume/smithi157-lv-4 created 2022-08-05T16:05:54.666 INFO:teuthology.orchestra.run.smithi059.stdout:persistentvolume/smithi201-lv-1 created 2022-08-05T16:05:54.709 INFO:teuthology.orchestra.run.smithi059.stdout:persistentvolume/smithi201-lv-2 created 2022-08-05T16:05:54.714 INFO:teuthology.orchestra.run.smithi059.stdout:persistentvolume/smithi201-lv-3 created 2022-08-05T16:05:54.766 INFO:teuthology.orchestra.run.smithi059.stdout:persistentvolume/smithi201-lv-4 created 2022-08-05T16:05:54.767 INFO:teuthology.orchestra.run.smithi059.stdout:storageclass.storage.k8s.io/scratch created 2022-08-05T16:05:54.769 DEBUG:teuthology.orchestra.run.smithi059:> kubectl taint node smithi059 node-role.kubernetes.io/master- || true 2022-08-05T16:05:54.917 INFO:teuthology.orchestra.run.smithi059.stdout:node/smithi059 untainted 2022-08-05T16:05:54.918 INFO:tasks.kubeadm:Kubeadm complete, yielding 2022-08-05T16:05:54.919 INFO:teuthology.run_tasks:Running task nvme_loop... 2022-08-05T16:05:54.983 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2022-08-05T16:05:55.026 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-08-05T16:05:55.027 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/scratch_devs of=/dev/stdout 2022-08-05T16:05:55.112 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-08-05T16:05:55.113 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-08-05T16:05:55.189 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-08-05T16:05:55.192 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-08-05T16:05:55.192 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 498 Links: 1 2022-08-05T16:05:55.193 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-08-05T16:05:55.208 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2022-08-05 16:05:51.776845600 +0000 2022-08-05T16:05:55.221 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2022-08-05 16:05:51.776845600 +0000 2022-08-05T16:05:55.234 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2022-08-05 16:05:51.776845600 +0000 2022-08-05T16:05:55.234 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2022-08-05T16:05:55.237 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2022-08-05T16:05:55.237 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2022-08-05T16:05:55.247 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000503367 s, 1.0 MB/s 2022-08-05T16:05:55.249 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-08-05T16:05:55.316 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-08-05T16:05:55.316 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-08-05T16:05:55.317 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 502 Links: 1 2022-08-05T16:05:55.317 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-08-05T16:05:55.318 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2022-08-05 16:05:51.908845597 +0000 2022-08-05T16:05:55.320 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2022-08-05 16:05:51.904845598 +0000 2022-08-05T16:05:55.320 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2022-08-05 16:05:51.904845598 +0000 2022-08-05T16:05:55.320 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2022-08-05T16:05:55.354 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2022-08-05T16:05:55.354 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2022-08-05T16:05:55.355 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000436042 s, 1.2 MB/s 2022-08-05T16:05:55.355 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-08-05T16:05:55.407 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-08-05T16:05:55.408 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-08-05T16:05:55.408 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 510 Links: 1 2022-08-05T16:05:55.409 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-08-05T16:05:55.409 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2022-08-05 16:05:52.052845594 +0000 2022-08-05T16:05:55.443 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2022-08-05 16:05:52.052845594 +0000 2022-08-05T16:05:55.446 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2022-08-05 16:05:52.052845594 +0000 2022-08-05T16:05:55.476 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2022-08-05T16:05:55.477 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2022-08-05T16:05:55.490 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2022-08-05T16:05:55.491 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000424969 s, 1.2 MB/s 2022-08-05T16:05:55.491 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-08-05T16:05:55.567 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-08-05T16:05:55.568 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-08-05T16:05:55.568 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 5h/5d Inode: 514 Links: 1 2022-08-05T16:05:55.569 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-08-05T16:05:55.569 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2022-08-05 16:05:52.132845592 +0000 2022-08-05T16:05:55.569 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2022-08-05 16:05:52.128845593 +0000 2022-08-05T16:05:55.641 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2022-08-05 16:05:52.128845593 +0000 2022-08-05T16:05:55.641 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2022-08-05T16:05:55.699 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2022-08-05T16:05:55.700 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2022-08-05T16:05:55.700 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000490746 s, 1.0 MB/s 2022-08-05T16:05:55.700 DEBUG:teuthology.orchestra.run.smithi059:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-08-05T16:05:56.047 INFO:teuthology.orchestra.run.smithi059.stdout:loop 2022-08-05T16:05:56.047 INFO:tasks.nvme_loop:Connecting nvme_loop smithi059:/dev/vg_nvme/lv_1... 2022-08-05T16:05:56.056 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-08-05T16:05:56.109 INFO:teuthology.orchestra.run.smithi059.stdout:1 2022-08-05T16:05:56.117 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/vg_nvme/lv_11 2022-08-05T16:05:56.135 INFO:tasks.nvme_loop:Connecting nvme_loop smithi059:/dev/vg_nvme/lv_2... 2022-08-05T16:05:56.151 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-08-05T16:05:56.239 INFO:teuthology.orchestra.run.smithi059.stdout:1 2022-08-05T16:05:56.258 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/vg_nvme/lv_21 2022-08-05T16:05:56.286 INFO:tasks.nvme_loop:Connecting nvme_loop smithi059:/dev/vg_nvme/lv_3... 2022-08-05T16:05:56.295 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-08-05T16:05:56.374 INFO:teuthology.orchestra.run.smithi059.stdout:1 2022-08-05T16:05:56.423 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/vg_nvme/lv_31 2022-08-05T16:05:56.462 INFO:tasks.nvme_loop:Connecting nvme_loop smithi059:/dev/vg_nvme/lv_4... 2022-08-05T16:05:56.463 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-08-05T16:05:56.503 INFO:teuthology.orchestra.run.smithi059.stdout:1 2022-08-05T16:05:56.559 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/vg_nvme/lv_41 2022-08-05T16:05:56.562 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-08-05T16:05:56.563 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/scratch_devs of=/dev/stdout 2022-08-05T16:05:56.706 DEBUG:teuthology.orchestra.run.smithi059:> sudo nvme list 2022-08-05T16:05:56.745 INFO:teuthology.orchestra.run.smithi059.stdout:Node SN Model Namespace Usage Format FW Rev 2022-08-05T16:05:56.746 INFO:teuthology.orchestra.run.smithi059.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-08-05T16:05:56.746 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/nvme0n1 CVFT5330009L400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-08-05T16:05:56.747 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/nvme1n1 9a8def51bfd032cbbf52 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-4 2022-08-05T16:05:56.747 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/nvme2n1 a867a02da59dbdc12a67 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-4 2022-08-05T16:05:56.747 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/nvme3n1 c5df6e555a0120665197 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-4 2022-08-05T16:05:56.760 INFO:teuthology.orchestra.run.smithi059.stdout:/dev/nvme4n1 a0801f3ce62e306dddfb Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-4 2022-08-05T16:05:56.761 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-08-05T16:05:56.761 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-08-05T16:05:56.762 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/scratch_devs 2022-08-05T16:05:56.958 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2022-08-05T16:05:56.958 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/scratch_devs of=/dev/stdout 2022-08-05T16:05:57.081 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-08-05T16:05:57.110 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-08-05T16:05:57.305 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-08-05T16:05:57.306 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-08-05T16:05:57.306 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 495 Links: 1 2022-08-05T16:05:57.306 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-08-05T16:05:57.308 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2022-08-05 16:05:52.257124348 +0000 2022-08-05T16:05:57.309 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2022-08-05 16:05:52.257124348 +0000 2022-08-05T16:05:57.310 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2022-08-05 16:05:52.257124348 +0000 2022-08-05T16:05:57.310 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2022-08-05T16:05:57.312 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2022-08-05T16:05:57.313 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2022-08-05T16:05:57.313 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000233957 s, 2.2 MB/s 2022-08-05T16:05:57.314 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-08-05T16:05:57.444 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-08-05T16:05:57.444 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-08-05T16:05:57.445 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 501 Links: 1 2022-08-05T16:05:57.445 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-08-05T16:05:57.445 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2022-08-05 16:05:52.365124346 +0000 2022-08-05T16:05:57.445 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2022-08-05 16:05:52.365124346 +0000 2022-08-05T16:05:57.445 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2022-08-05 16:05:52.365124346 +0000 2022-08-05T16:05:57.581 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2022-08-05T16:05:57.582 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2022-08-05T16:05:57.583 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2022-08-05T16:05:57.583 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000326875 s, 1.6 MB/s 2022-08-05T16:05:57.598 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-08-05T16:05:57.853 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-08-05T16:05:57.853 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-08-05T16:05:57.853 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 504 Links: 1 2022-08-05T16:05:57.854 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-08-05T16:05:57.854 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2022-08-05 16:05:52.453124344 +0000 2022-08-05T16:05:57.854 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2022-08-05 16:05:52.453124344 +0000 2022-08-05T16:05:57.927 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2022-08-05 16:05:52.453124344 +0000 2022-08-05T16:05:57.927 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2022-08-05T16:05:57.983 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2022-08-05T16:05:57.983 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2022-08-05T16:05:57.983 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000247444 s, 2.1 MB/s 2022-08-05T16:05:57.984 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-08-05T16:05:58.116 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-08-05T16:05:58.117 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-08-05T16:05:58.117 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 513 Links: 1 2022-08-05T16:05:58.147 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-08-05T16:05:58.171 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2022-08-05 16:05:52.561124342 +0000 2022-08-05T16:05:58.172 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2022-08-05 16:05:52.561124342 +0000 2022-08-05T16:05:58.182 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2022-08-05 16:05:52.561124342 +0000 2022-08-05T16:05:58.182 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2022-08-05T16:05:58.183 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2022-08-05T16:05:58.184 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2022-08-05T16:05:58.184 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000222512 s, 2.3 MB/s 2022-08-05T16:05:58.210 DEBUG:teuthology.orchestra.run.smithi157:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-08-05T16:05:58.403 INFO:teuthology.orchestra.run.smithi157.stdout:loop 2022-08-05T16:05:58.404 INFO:tasks.nvme_loop:Connecting nvme_loop smithi157:/dev/vg_nvme/lv_1... 2022-08-05T16:05:58.404 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-08-05T16:05:58.521 INFO:teuthology.orchestra.run.smithi157.stdout:1 2022-08-05T16:05:58.539 INFO:teuthology.orchestra.run.smithi157.stdout:/dev/vg_nvme/lv_11 2022-08-05T16:05:58.540 INFO:tasks.nvme_loop:Connecting nvme_loop smithi157:/dev/vg_nvme/lv_2... 2022-08-05T16:05:58.540 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-08-05T16:05:58.608 INFO:teuthology.orchestra.run.smithi157.stdout:1 2022-08-05T16:05:58.714 INFO:teuthology.orchestra.run.smithi157.stdout:/dev/vg_nvme/lv_21 2022-08-05T16:05:58.787 INFO:tasks.nvme_loop:Connecting nvme_loop smithi157:/dev/vg_nvme/lv_3... 2022-08-05T16:05:58.788 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-08-05T16:05:58.976 INFO:teuthology.orchestra.run.smithi157.stdout:1 2022-08-05T16:05:58.976 INFO:teuthology.orchestra.run.smithi157.stdout:/dev/vg_nvme/lv_31 2022-08-05T16:05:59.052 INFO:tasks.nvme_loop:Connecting nvme_loop smithi157:/dev/vg_nvme/lv_4... 2022-08-05T16:05:59.053 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-08-05T16:05:59.108 INFO:teuthology.orchestra.run.smithi157.stdout:1 2022-08-05T16:05:59.135 INFO:teuthology.orchestra.run.smithi157.stdout:/dev/vg_nvme/lv_41 2022-08-05T16:05:59.160 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2022-08-05T16:05:59.160 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/scratch_devs of=/dev/stdout 2022-08-05T16:05:59.247 DEBUG:teuthology.orchestra.run.smithi157:> sudo nvme list 2022-08-05T16:05:59.263 INFO:teuthology.orchestra.run.smithi157.stdout:Node SN Model Namespace Usage Format FW Rev 2022-08-05T16:05:59.294 INFO:teuthology.orchestra.run.smithi157.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-08-05T16:05:59.295 INFO:teuthology.orchestra.run.smithi157.stdout:/dev/nvme0n1 PHMB7476002V480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2022-08-05T16:05:59.295 INFO:teuthology.orchestra.run.smithi157.stdout:/dev/nvme1n1 563258dbf090786607e4 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-4 2022-08-05T16:05:59.295 INFO:teuthology.orchestra.run.smithi157.stdout:/dev/nvme2n1 7c694fd215e1fcd44600 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-4 2022-08-05T16:05:59.295 INFO:teuthology.orchestra.run.smithi157.stdout:/dev/nvme3n1 f510b443dab8811dbf12 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-4 2022-08-05T16:05:59.307 INFO:teuthology.orchestra.run.smithi157.stdout:/dev/nvme4n1 4758a7329a05006a3b37 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-4 2022-08-05T16:05:59.347 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-08-05T16:05:59.374 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2022-08-05T16:05:59.375 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/scratch_devs 2022-08-05T16:05:59.632 DEBUG:teuthology.orchestra.run.smithi201:> set -ex 2022-08-05T16:05:59.633 DEBUG:teuthology.orchestra.run.smithi201:> dd if=/scratch_devs of=/dev/stdout 2022-08-05T16:05:59.834 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-08-05T16:05:59.834 DEBUG:teuthology.orchestra.run.smithi201:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-08-05T16:06:00.245 INFO:teuthology.orchestra.run.smithi201.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-08-05T16:06:00.289 INFO:teuthology.orchestra.run.smithi201.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-08-05T16:06:00.306 INFO:teuthology.orchestra.run.smithi201.stdout:Device: 5h/5d Inode: 493 Links: 1 2022-08-05T16:06:00.306 INFO:teuthology.orchestra.run.smithi201.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-08-05T16:06:00.307 INFO:teuthology.orchestra.run.smithi201.stdout:Access: 2022-08-05 16:05:52.679351387 +0000 2022-08-05T16:06:00.307 INFO:teuthology.orchestra.run.smithi201.stdout:Modify: 2022-08-05 16:05:52.679351387 +0000 2022-08-05T16:06:00.307 INFO:teuthology.orchestra.run.smithi201.stdout:Change: 2022-08-05 16:05:52.679351387 +0000 2022-08-05T16:06:00.308 INFO:teuthology.orchestra.run.smithi201.stdout: Birth: - 2022-08-05T16:06:00.309 INFO:teuthology.orchestra.run.smithi201.stderr:1+0 records in 2022-08-05T16:06:00.309 INFO:teuthology.orchestra.run.smithi201.stderr:1+0 records out 2022-08-05T16:06:00.368 INFO:teuthology.orchestra.run.smithi201.stderr:512 bytes copied, 0.000586989 s, 872 kB/s 2022-08-05T16:06:00.369 DEBUG:teuthology.orchestra.run.smithi201:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-08-05T16:06:00.448 INFO:teuthology.orchestra.run.smithi201.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-08-05T16:06:00.448 INFO:teuthology.orchestra.run.smithi201.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-08-05T16:06:00.448 INFO:teuthology.orchestra.run.smithi201.stdout:Device: 5h/5d Inode: 501 Links: 1 2022-08-05T16:06:00.449 INFO:teuthology.orchestra.run.smithi201.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-08-05T16:06:00.449 INFO:teuthology.orchestra.run.smithi201.stdout:Access: 2022-08-05 16:05:52.791351385 +0000 2022-08-05T16:06:00.449 INFO:teuthology.orchestra.run.smithi201.stdout:Modify: 2022-08-05 16:05:52.791351385 +0000 2022-08-05T16:06:00.449 INFO:teuthology.orchestra.run.smithi201.stdout:Change: 2022-08-05 16:05:52.791351385 +0000 2022-08-05T16:06:00.561 INFO:teuthology.orchestra.run.smithi201.stdout: Birth: - 2022-08-05T16:06:00.627 INFO:teuthology.orchestra.run.smithi201.stderr:1+0 records in 2022-08-05T16:06:00.627 INFO:teuthology.orchestra.run.smithi201.stderr:1+0 records out 2022-08-05T16:06:00.684 INFO:teuthology.orchestra.run.smithi201.stderr:512 bytes copied, 0.000419045 s, 1.2 MB/s 2022-08-05T16:06:00.773 DEBUG:teuthology.orchestra.run.smithi201:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-08-05T16:06:00.936 INFO:teuthology.orchestra.run.smithi201.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-08-05T16:06:00.937 INFO:teuthology.orchestra.run.smithi201.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-08-05T16:06:00.937 INFO:teuthology.orchestra.run.smithi201.stdout:Device: 5h/5d Inode: 504 Links: 1 2022-08-05T16:06:00.938 INFO:teuthology.orchestra.run.smithi201.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-08-05T16:06:00.938 INFO:teuthology.orchestra.run.smithi201.stdout:Access: 2022-08-05 16:05:52.911351382 +0000 2022-08-05T16:06:00.939 INFO:teuthology.orchestra.run.smithi201.stdout:Modify: 2022-08-05 16:05:52.911351382 +0000 2022-08-05T16:06:00.939 INFO:teuthology.orchestra.run.smithi201.stdout:Change: 2022-08-05 16:05:52.911351382 +0000 2022-08-05T16:06:00.939 INFO:teuthology.orchestra.run.smithi201.stdout: Birth: - 2022-08-05T16:06:00.941 INFO:teuthology.orchestra.run.smithi201.stderr:1+0 records in 2022-08-05T16:06:00.947 INFO:teuthology.orchestra.run.smithi201.stderr:1+0 records out 2022-08-05T16:06:00.948 INFO:teuthology.orchestra.run.smithi201.stderr:512 bytes copied, 0.000508657 s, 1.0 MB/s 2022-08-05T16:06:00.949 DEBUG:teuthology.orchestra.run.smithi201:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-08-05T16:06:01.279 INFO:teuthology.orchestra.run.smithi201.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-08-05T16:06:01.280 INFO:teuthology.orchestra.run.smithi201.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-08-05T16:06:01.280 INFO:teuthology.orchestra.run.smithi201.stdout:Device: 5h/5d Inode: 513 Links: 1 2022-08-05T16:06:01.280 INFO:teuthology.orchestra.run.smithi201.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-08-05T16:06:01.281 INFO:teuthology.orchestra.run.smithi201.stdout:Access: 2022-08-05 16:05:53.027351380 +0000 2022-08-05T16:06:01.281 INFO:teuthology.orchestra.run.smithi201.stdout:Modify: 2022-08-05 16:05:53.027351380 +0000 2022-08-05T16:06:01.281 INFO:teuthology.orchestra.run.smithi201.stdout:Change: 2022-08-05 16:05:53.027351380 +0000 2022-08-05T16:06:01.328 INFO:teuthology.orchestra.run.smithi201.stdout: Birth: - 2022-08-05T16:06:01.444 INFO:teuthology.orchestra.run.smithi201.stderr:1+0 records in 2022-08-05T16:06:01.456 INFO:teuthology.orchestra.run.smithi201.stderr:1+0 records out 2022-08-05T16:06:01.456 INFO:teuthology.orchestra.run.smithi201.stderr:512 bytes copied, 0.000490864 s, 1.0 MB/s 2022-08-05T16:06:01.470 DEBUG:teuthology.orchestra.run.smithi201:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2022-08-05T16:06:01.679 INFO:teuthology.orchestra.run.smithi201.stdout:loop 2022-08-05T16:06:01.680 INFO:tasks.nvme_loop:Connecting nvme_loop smithi201:/dev/vg_nvme/lv_1... 2022-08-05T16:06:01.681 DEBUG:teuthology.orchestra.run.smithi201:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2022-08-05T16:06:01.878 INFO:teuthology.orchestra.run.smithi201.stdout:1 2022-08-05T16:06:01.883 INFO:teuthology.orchestra.run.smithi201.stdout:/dev/vg_nvme/lv_11 2022-08-05T16:06:01.957 INFO:tasks.nvme_loop:Connecting nvme_loop smithi201:/dev/vg_nvme/lv_2... 2022-08-05T16:06:02.022 DEBUG:teuthology.orchestra.run.smithi201:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2022-08-05T16:06:02.065 INFO:teuthology.orchestra.run.smithi201.stdout:1 2022-08-05T16:06:02.095 INFO:teuthology.orchestra.run.smithi201.stdout:/dev/vg_nvme/lv_21 2022-08-05T16:06:02.135 INFO:tasks.nvme_loop:Connecting nvme_loop smithi201:/dev/vg_nvme/lv_3... 2022-08-05T16:06:02.136 DEBUG:teuthology.orchestra.run.smithi201:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2022-08-05T16:06:02.508 INFO:teuthology.orchestra.run.smithi201.stdout:1 2022-08-05T16:06:02.581 INFO:teuthology.orchestra.run.smithi201.stdout:/dev/vg_nvme/lv_31 2022-08-05T16:06:02.581 INFO:tasks.nvme_loop:Connecting nvme_loop smithi201:/dev/vg_nvme/lv_4... 2022-08-05T16:06:02.582 DEBUG:teuthology.orchestra.run.smithi201:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2022-08-05T16:06:02.788 INFO:teuthology.orchestra.run.smithi201.stdout:1 2022-08-05T16:06:02.789 INFO:teuthology.orchestra.run.smithi201.stdout:/dev/vg_nvme/lv_41 2022-08-05T16:06:02.790 DEBUG:teuthology.orchestra.run.smithi201:> set -ex 2022-08-05T16:06:02.906 DEBUG:teuthology.orchestra.run.smithi201:> dd if=/scratch_devs of=/dev/stdout 2022-08-05T16:06:03.111 DEBUG:teuthology.orchestra.run.smithi201:> sudo nvme list 2022-08-05T16:06:03.172 INFO:teuthology.orchestra.run.smithi201.stdout:Node SN Model Namespace Usage Format FW Rev 2022-08-05T16:06:03.172 INFO:teuthology.orchestra.run.smithi201.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2022-08-05T16:06:03.288 INFO:teuthology.orchestra.run.smithi201.stdout:/dev/nvme0n1 CVFT626500CN400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2022-08-05T16:06:03.288 INFO:teuthology.orchestra.run.smithi201.stdout:/dev/nvme1n1 88fc1eb6bb969d4713c0 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-4 2022-08-05T16:06:03.289 INFO:teuthology.orchestra.run.smithi201.stdout:/dev/nvme2n1 82a39a73ef874e589b4b Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-4 2022-08-05T16:06:03.289 INFO:teuthology.orchestra.run.smithi201.stdout:/dev/nvme3n1 9d6cf81975af05464fb5 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-4 2022-08-05T16:06:03.292 INFO:teuthology.orchestra.run.smithi201.stdout:/dev/nvme4n1 15149f0d0023efa75566 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.15.0-4 2022-08-05T16:06:03.294 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2022-08-05T16:06:03.295 DEBUG:teuthology.orchestra.run.smithi201:> set -ex 2022-08-05T16:06:03.295 DEBUG:teuthology.orchestra.run.smithi201:> sudo dd of=/scratch_devs 2022-08-05T16:06:03.395 INFO:teuthology.run_tasks:Running task rook... 2022-08-05T16:06:05.169 INFO:tasks.rook:Rook start 2022-08-05T16:06:05.225 INFO:tasks.rook:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '4a7e6c4ac29de794744a2ce76ca2b36bc637894d', 'rook_image': 'rook/ceph:master'} 2022-08-05T16:06:05.227 INFO:tasks.rook:Ceph image is quay.ceph.io/ceph-ci/ceph:4a7e6c4ac29de794744a2ce76ca2b36bc637894d 2022-08-05T16:06:05.256 INFO:tasks.rook:Cloning https://github.com/rook/rook branch master 2022-08-05T16:06:05.257 DEBUG:teuthology.orchestra.run.smithi059:> rm -rf rook && git clone --single-branch --branch master https://github.com/rook/rook rook 2022-08-05T16:06:05.332 INFO:teuthology.orchestra.run.smithi059.stderr:Cloning into 'rook'... 2022-08-05T16:06:08.988 INFO:tasks.rook:/home/teuthworker 2022-08-05T16:06:08.992 DEBUG:teuthology.orchestra.run.smithi059:> PATH=/sbin:/usr/sbin ip addr show 2022-08-05T16:06:09.024 INFO:teuthology.orchestra.run.smithi059.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-08-05T16:06:09.025 INFO:teuthology.orchestra.run.smithi059.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-08-05T16:06:09.025 INFO:teuthology.orchestra.run.smithi059.stdout: inet 127.0.0.1/8 scope host lo 2022-08-05T16:06:09.043 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2022-08-05T16:06:09.051 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 ::1/128 scope host 2022-08-05T16:06:09.052 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2022-08-05T16:06:09.052 INFO:teuthology.orchestra.run.smithi059.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-08-05T16:06:09.053 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:ca brd ff:ff:ff:ff:ff:ff 2022-08-05T16:06:09.053 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f0 2022-08-05T16:06:09.054 INFO:teuthology.orchestra.run.smithi059.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2022-08-05T16:06:09.106 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2a brd ff:ff:ff:ff:ff:ff 2022-08-05T16:06:09.107 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f0 2022-08-05T16:06:09.107 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.21.15.59/20 brd 172.21.15.255 scope global dynamic ens1f0 2022-08-05T16:06:09.108 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft 42373sec preferred_lft 42373sec 2022-08-05T16:06:09.108 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::ec4:7aff:febd:192a/64 scope link 2022-08-05T16:06:09.108 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2022-08-05T16:06:09.109 INFO:teuthology.orchestra.run.smithi059.stdout:4: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-08-05T16:06:09.109 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:6b:fe:cb brd ff:ff:ff:ff:ff:ff 2022-08-05T16:06:09.112 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp7s0f1 2022-08-05T16:06:09.116 INFO:teuthology.orchestra.run.smithi059.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2022-08-05T16:06:09.155 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 0c:c4:7a:bd:19:2b brd ff:ff:ff:ff:ff:ff 2022-08-05T16:06:09.156 INFO:teuthology.orchestra.run.smithi059.stdout: altname enp1s0f1 2022-08-05T16:06:09.160 INFO:teuthology.orchestra.run.smithi059.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2022-08-05T16:06:09.178 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 02:42:18:52:ec:6e brd ff:ff:ff:ff:ff:ff 2022-08-05T16:06:09.179 INFO:teuthology.orchestra.run.smithi059.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2022-08-05T16:06:09.179 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2022-08-05T16:06:09.180 INFO:teuthology.orchestra.run.smithi059.stdout:7: flannel.1: mtu 1450 qdisc noqueue state UNKNOWN group default 2022-08-05T16:06:09.180 INFO:teuthology.orchestra.run.smithi059.stdout: link/ether 4e:2b:f7:5f:9c:1b brd ff:ff:ff:ff:ff:ff 2022-08-05T16:06:09.198 INFO:teuthology.orchestra.run.smithi059.stdout: inet 10.249.208.0/32 scope global flannel.1 2022-08-05T16:06:09.198 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2022-08-05T16:06:09.198 INFO:teuthology.orchestra.run.smithi059.stdout: inet6 fe80::4c2b:f7ff:fe5f:9c1b/64 scope link 2022-08-05T16:06:09.221 INFO:teuthology.orchestra.run.smithi059.stdout: valid_lft forever preferred_lft forever 2022-08-05T16:06:09.249 DEBUG:teuthology.orchestra.run.smithi059:> which systemctl 2022-08-05T16:06:09.268 INFO:teuthology.orchestra.run.smithi059.stdout:/usr/bin/systemctl 2022-08-05T16:06:09.269 DEBUG:teuthology.orchestra.run.smithi059:> true 2022-08-05T16:06:09.425 DEBUG:teuthology.orchestra.run.smithi059:> sudo lsb_release -is 2022-08-05T16:06:09.564 INFO:teuthology.orchestra.run.smithi059.stdout:Ubuntu 2022-08-05T16:06:09.565 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-08-05T16:06:09.585 INFO:tasks.rook:['__class__', '__del__', '__delattr__', '__dir__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_format_size', '_reconnect', '_runner', '_set_iface_and_cidr', '_sftp_get_file', '_sftp_get_size', '_sftp_open_file', '_sftp_put_file', 'chcon', 'chmod', 'connect', 'copy_file', 'ensure_online', 'get_file', 'get_tar', 'get_tar_stream', 'mkdtemp', 'mktemp', 'move_file', 'put_file', 'read_file', 'reconnect', 'remove', 'run', 'sh', 'sh_file', 'sudo_write_file', 'write_file'] 2022-08-05T16:06:09.636 DEBUG:teuthology.orchestra.remote:smithi059:rook/deploy/examples/operator.yaml is 24KB 2022-08-05T16:06:09.885 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-08-05T16:06:09.919 DEBUG:teuthology.orchestra.run.smithi059:> dd if=rook/deploy/examples/operator.yaml of=/dev/stdout 2022-08-05T16:06:10.072 INFO:tasks.rook:Patching operator to use image rook/ceph:master 2022-08-05T16:06:10.828 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-08-05T16:06:10.831 DEBUG:teuthology.orchestra.run.smithi059:> dd of=operator.yaml 2022-08-05T16:06:10.939 INFO:tasks.rook:Deploying operator 2022-08-05T16:06:11.027 DEBUG:teuthology.orchestra.remote:smithi059:rook/deploy/examples/operator.yaml is 24KB 2022-08-05T16:06:11.715 DEBUG:teuthology.orchestra.remote:smithi059:rook/deploy/examples/operator.yaml is 24KB 2022-08-05T16:06:12.088 DEBUG:teuthology.orchestra.run.smithi059:> kubectl create -f rook/deploy/examples/crds.yaml -f rook/deploy/examples/common.yaml -f operator.yaml 2022-08-05T16:06:12.391 INFO:teuthology.orchestra.run.smithi059.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpoolradosnamespaces.ceph.rook.io created 2022-08-05T16:06:12.415 INFO:teuthology.orchestra.run.smithi059.stdout:customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created 2022-08-05T16:06:12.446 INFO:teuthology.orchestra.run.smithi059.stdout:customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created 2022-08-05T16:06:12.526 INFO:teuthology.orchestra.run.smithi059.stdout:customresourcedefinition.apiextensions.k8s.io/cephbuckettopics.ceph.rook.io created 2022-08-05T16:06:12.558 INFO:teuthology.orchestra.run.smithi059.stdout:customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created 2022-08-05T16:06:12.759 INFO:teuthology.orchestra.run.smithi059.stdout:customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created 2022-08-05T16:06:12.787 INFO:teuthology.orchestra.run.smithi059.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created 2022-08-05T16:06:12.919 INFO:teuthology.orchestra.run.smithi059.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created 2022-08-05T16:06:12.919 INFO:teuthology.orchestra.run.smithi059.stdout:customresourcedefinition.apiextensions.k8s.io/cephfilesystemsubvolumegroups.ceph.rook.io created 2022-08-05T16:06:12.969 INFO:teuthology.orchestra.run.smithi059.stdout:customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created 2022-08-05T16:06:13.002 INFO:teuthology.orchestra.run.smithi059.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created 2022-08-05T16:06:13.104 INFO:teuthology.orchestra.run.smithi059.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created 2022-08-05T16:06:13.124 INFO:teuthology.orchestra.run.smithi059.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created 2022-08-05T16:06:13.150 INFO:teuthology.orchestra.run.smithi059.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created 2022-08-05T16:06:13.187 INFO:teuthology.orchestra.run.smithi059.stdout:customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created 2022-08-05T16:06:13.250 INFO:teuthology.orchestra.run.smithi059.stdout:customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created 2022-08-05T16:06:13.264 INFO:teuthology.orchestra.run.smithi059.stdout:customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created 2022-08-05T16:06:13.316 INFO:teuthology.orchestra.run.smithi059.stdout:customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created 2022-08-05T16:06:13.353 INFO:teuthology.orchestra.run.smithi059.stdout:namespace/rook-ceph created 2022-08-05T16:06:13.452 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created 2022-08-05T16:06:13.453 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created 2022-08-05T16:06:13.472 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrole.rbac.authorization.k8s.io/psp:rook created 2022-08-05T16:06:13.504 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-08-05T16:06:13.521 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created 2022-08-05T16:06:13.523 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-08-05T16:06:13.524 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-global created 2022-08-05T16:06:13.572 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-08-05T16:06:13.573 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-08-05T16:06:13.641 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-08-05T16:06:13.677 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created 2022-08-05T16:06:13.680 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrole.rbac.authorization.k8s.io/rook-ceph-system created 2022-08-05T16:06:13.680 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created 2022-08-05T16:06:13.681 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-08-05T16:06:13.682 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created 2022-08-05T16:06:13.682 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created 2022-08-05T16:06:13.683 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created 2022-08-05T16:06:13.716 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created 2022-08-05T16:06:13.729 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-08-05T16:06:13.734 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-08-05T16:06:13.792 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created 2022-08-05T16:06:13.793 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created 2022-08-05T16:06:13.794 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created 2022-08-05T16:06:13.795 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created 2022-08-05T16:06:13.817 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created 2022-08-05T16:06:13.892 INFO:teuthology.orchestra.run.smithi059.stderr:Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ 2022-08-05T16:06:13.893 INFO:teuthology.orchestra.run.smithi059.stdout:podsecuritypolicy.policy/00-rook-privileged created 2022-08-05T16:06:13.893 INFO:teuthology.orchestra.run.smithi059.stdout:role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created 2022-08-05T16:06:13.977 INFO:teuthology.orchestra.run.smithi059.stdout:role.rbac.authorization.k8s.io/rbd-csi-nodeplugin created 2022-08-05T16:06:13.977 INFO:teuthology.orchestra.run.smithi059.stdout:role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created 2022-08-05T16:06:13.978 INFO:teuthology.orchestra.run.smithi059.stdout:role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-08-05T16:06:14.009 INFO:teuthology.orchestra.run.smithi059.stdout:role.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-08-05T16:06:14.011 INFO:teuthology.orchestra.run.smithi059.stdout:role.rbac.authorization.k8s.io/rook-ceph-osd created 2022-08-05T16:06:14.012 INFO:teuthology.orchestra.run.smithi059.stdout:role.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-08-05T16:06:14.012 INFO:teuthology.orchestra.run.smithi059.stdout:role.rbac.authorization.k8s.io/rook-ceph-rgw created 2022-08-05T16:06:14.012 INFO:teuthology.orchestra.run.smithi059.stdout:role.rbac.authorization.k8s.io/rook-ceph-system created 2022-08-05T16:06:14.013 INFO:teuthology.orchestra.run.smithi059.stdout:rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created 2022-08-05T16:06:14.013 INFO:teuthology.orchestra.run.smithi059.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-role-cfg created 2022-08-05T16:06:14.015 INFO:teuthology.orchestra.run.smithi059.stdout:rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created 2022-08-05T16:06:14.106 INFO:teuthology.orchestra.run.smithi059.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created 2022-08-05T16:06:14.116 INFO:teuthology.orchestra.run.smithi059.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created 2022-08-05T16:06:14.117 INFO:teuthology.orchestra.run.smithi059.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created 2022-08-05T16:06:14.117 INFO:teuthology.orchestra.run.smithi059.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created 2022-08-05T16:06:14.122 INFO:teuthology.orchestra.run.smithi059.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created 2022-08-05T16:06:14.194 INFO:teuthology.orchestra.run.smithi059.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created 2022-08-05T16:06:14.195 INFO:teuthology.orchestra.run.smithi059.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created 2022-08-05T16:06:14.231 INFO:teuthology.orchestra.run.smithi059.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created 2022-08-05T16:06:14.239 INFO:teuthology.orchestra.run.smithi059.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created 2022-08-05T16:06:14.329 INFO:teuthology.orchestra.run.smithi059.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created 2022-08-05T16:06:14.338 INFO:teuthology.orchestra.run.smithi059.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd-psp created 2022-08-05T16:06:14.339 INFO:teuthology.orchestra.run.smithi059.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-rgw created 2022-08-05T16:06:14.339 INFO:teuthology.orchestra.run.smithi059.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-rgw-psp created 2022-08-05T16:06:14.339 INFO:teuthology.orchestra.run.smithi059.stdout:rolebinding.rbac.authorization.k8s.io/rook-ceph-system created 2022-08-05T16:06:14.340 INFO:teuthology.orchestra.run.smithi059.stdout:serviceaccount/rook-ceph-cmd-reporter created 2022-08-05T16:06:14.340 INFO:teuthology.orchestra.run.smithi059.stdout:serviceaccount/rook-ceph-mgr created 2022-08-05T16:06:14.341 INFO:teuthology.orchestra.run.smithi059.stdout:serviceaccount/rook-ceph-osd created 2022-08-05T16:06:14.341 INFO:teuthology.orchestra.run.smithi059.stdout:serviceaccount/rook-ceph-purge-osd created 2022-08-05T16:06:14.341 INFO:teuthology.orchestra.run.smithi059.stdout:serviceaccount/rook-ceph-rgw created 2022-08-05T16:06:14.462 INFO:teuthology.orchestra.run.smithi059.stdout:serviceaccount/rook-ceph-system created 2022-08-05T16:06:14.462 INFO:teuthology.orchestra.run.smithi059.stdout:serviceaccount/rook-csi-cephfs-plugin-sa created 2022-08-05T16:06:14.463 INFO:teuthology.orchestra.run.smithi059.stdout:serviceaccount/rook-csi-cephfs-provisioner-sa created 2022-08-05T16:06:14.463 INFO:teuthology.orchestra.run.smithi059.stdout:serviceaccount/rook-csi-rbd-plugin-sa created 2022-08-05T16:06:14.463 INFO:teuthology.orchestra.run.smithi059.stdout:serviceaccount/rook-csi-rbd-provisioner-sa created 2022-08-05T16:06:14.464 INFO:teuthology.orchestra.run.smithi059.stdout:configmap/rook-ceph-operator-config created 2022-08-05T16:06:14.480 INFO:teuthology.orchestra.run.smithi059.stdout:deployment.apps/rook-ceph-operator created 2022-08-05T16:06:14.481 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-08-05T16:06:14.571 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:06:14.571 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 0/1 ContainerCreating 0 0s 2022-08-05T16:06:24.573 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-08-05T16:06:24.670 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:06:24.671 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 0/1 ContainerCreating 0 10s 2022-08-05T16:06:34.680 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-08-05T16:06:34.766 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:06:34.767 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 0/1 ContainerCreating 0 20s 2022-08-05T16:06:44.774 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-08-05T16:06:44.884 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:06:44.884 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 0/1 ContainerCreating 0 30s 2022-08-05T16:06:54.886 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-operator 2022-08-05T16:06:54.982 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:06:54.982 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 40s 2022-08-05T16:06:54.998 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph logs -f rook-ceph-operator-b5c96c99b-5hsx5 2022-08-05T16:06:55.107 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-08-05T16:06:55.108 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/scratch_devs of=/dev/stdout 2022-08-05T16:06:55.121 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.403192 I | rookcmd: starting Rook v1.9.0-alpha.0.484.g03a9ebdc5 with arguments '/usr/local/bin/rook ceph operator' 2022-08-05T16:06:55.179 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.403304 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO, --operator-image=, --service-account= 2022-08-05T16:06:55.180 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.403309 I | cephcmd: starting Rook-Ceph operator 2022-08-05T16:06:55.180 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.555876 I | cephcmd: base ceph version inside the rook operator image is "ceph version 17.2.1 (ec95624474b1871a821a912b8c3af68f8f8e7aa1) quincy (stable)" 2022-08-05T16:06:55.180 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.563487 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-08-05T16:06:55.181 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.563505 I | operator: watching all namespaces for Ceph CRs 2022-08-05T16:06:55.181 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.563556 I | operator: setting up schemes 2022-08-05T16:06:55.181 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.566428 I | operator: setting up the controller-runtime manager 2022-08-05T16:06:55.181 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.770331 I | operator: Fetching webhook cert-manager-webhook to see if cert-manager is installed. 2022-08-05T16:06:55.205 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.773032 I | operator: failed to get cert manager 2022-08-05T16:06:55.215 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.773097 I | ceph-cluster-controller: successfully started 2022-08-05T16:06:55.222 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.773230 I | ceph-cluster-controller: enabling hotplug orchestration 2022-08-05T16:06:55.223 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.773268 I | ceph-crashcollector-controller: successfully started 2022-08-05T16:06:55.223 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.773305 I | ceph-block-pool-controller: successfully started 2022-08-05T16:06:55.223 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.773339 I | ceph-object-store-user-controller: successfully started 2022-08-05T16:06:55.223 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.773369 I | ceph-object-realm-controller: successfully started 2022-08-05T16:06:55.223 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.773399 I | ceph-object-zonegroup-controller: successfully started 2022-08-05T16:06:55.224 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.773416 I | ceph-object-zone-controller: successfully started 2022-08-05T16:06:55.224 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.773591 I | ceph-object-controller: successfully started 2022-08-05T16:06:55.224 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.773641 I | ceph-file-controller: successfully started 2022-08-05T16:06:55.260 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.773692 I | ceph-nfs-controller: successfully started 2022-08-05T16:06:55.261 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.773747 I | ceph-rbd-mirror-controller: successfully started 2022-08-05T16:06:55.262 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.773795 I | ceph-client-controller: successfully started 2022-08-05T16:06:55.283 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.773848 I | ceph-filesystem-mirror-controller: successfully started 2022-08-05T16:06:55.312 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.773896 I | operator: rook-ceph-operator-config-controller successfully started 2022-08-05T16:06:55.312 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.773931 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-08-05T16:06:55.313 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.773967 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-08-05T16:06:55.313 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.773991 I | ceph-bucket-topic: successfully started 2022-08-05T16:06:55.313 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.774020 I | ceph-bucket-notification: successfully started 2022-08-05T16:06:55.319 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.774053 I | ceph-bucket-notification: successfully started 2022-08-05T16:06:55.319 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.774085 I | ceph-fs-subvolumegroup-controller: successfully started 2022-08-05T16:06:55.320 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.774109 I | blockpool-rados-namespace-controller: successfully started 2022-08-05T16:06:55.320 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.775268 I | operator: starting the controller-runtime manager 2022-08-05T16:06:55.321 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.877824 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-08-05T16:06:55.321 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.877871 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-08-05T16:06:55.338 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.877889 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-08-05T16:06:55.339 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.880311 I | ceph-csi: CSI Ceph RBD driver disabled 2022-08-05T16:06:55.340 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.880350 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists 2022-08-05T16:06:55.341 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.900925 I | operator: rook-ceph-operator-config-controller done reconciling 2022-08-05T16:06:55.341 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.900969 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists 2022-08-05T16:06:55.342 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.907904 I | ceph-csi: successfully removed CSI Ceph RBD driver 2022-08-05T16:06:55.342 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.907926 I | ceph-csi: CSI CephFS driver disabled 2022-08-05T16:06:55.342 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.907935 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists 2022-08-05T16:06:55.343 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.910010 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists 2022-08-05T16:06:55.343 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.916100 I | ceph-csi: successfully removed CSI CephFS driver 2022-08-05T16:06:55.344 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.916118 I | ceph-csi: CSI NFS driver disabled 2022-08-05T16:06:55.378 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.916125 I | op-k8sutil: removing daemonset csi-nfsplugin if it exists 2022-08-05T16:06:55.379 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.918272 I | op-k8sutil: removing deployment csi-nfsplugin-provisioner if it exists 2022-08-05T16:06:55.379 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:51.924999 I | ceph-csi: successfully removed CSI NFS driver 2022-08-05T16:06:55.417 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2022-08-05T16:06:55.417 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/scratch_devs of=/dev/stdout 2022-08-05T16:06:55.440 DEBUG:teuthology.orchestra.run.smithi201:> set -ex 2022-08-05T16:06:55.440 DEBUG:teuthology.orchestra.run.smithi201:> dd if=/scratch_devs of=/dev/stdout 2022-08-05T16:06:55.480 INFO:tasks.rook: override: [mgr] debug mgr = 20 2022-08-05T16:06:55.481 INFO:tasks.rook: override: [mgr] debug ms = 1 2022-08-05T16:06:55.482 INFO:tasks.rook: override: [mon] debug mon = 20 2022-08-05T16:06:55.482 INFO:tasks.rook: override: [mon] debug ms = 1 2022-08-05T16:06:55.524 INFO:tasks.rook: override: [mon] debug paxos = 20 2022-08-05T16:06:55.525 INFO:tasks.rook: override: [osd] debug ms = 1 2022-08-05T16:06:55.525 INFO:tasks.rook: override: [osd] debug osd = 20 2022-08-05T16:06:55.528 INFO:tasks.rook:Config: b"[global]\n\nlog to file = true\n\nmon clock drift allowed = 1.000\n\n# replicate across OSDs, not hosts\nosd crush chooseleaf type = 0\n\n# enable some debugging\nauth debug = true\nms die on old message = true\nms die on bug = true\ndebug asserts on shutdown = true\n\n\n[osd]\n# debugging\nosd debug shutdown = true\nosd debug op order = true\nosd debug verify stray on activate = true\nosd debug pg log writeout = true\nosd debug verify cached snaps = true\nosd debug verify missing on start = true\nosd debug misdirected ops = true\nosd op queue = debug_random\nosd op queue cut off = debug_random\nosd shutdown pgref assert = true\nbdev debug aio = true\nosd sloppy crc = true\ndebug ms = 1\ndebug osd = 20\n\n\n[mon]\n# rotate auth tickets quickly to exercise renewal paths\nauth mon ticket ttl = 660# 11m\nauth service ticket ttl = 240# 4m\n\n# don't complain about global id reclaim\nmon_warn_on_insecure_global_id_reclaim = false\nmon_warn_on_insecure_global_id_reclaim_allowed = false\ndebug mon = 20\ndebug ms = 1\ndebug paxos = 20\n[mgr]\ndebug mgr = 20\ndebug ms = 1\n" 2022-08-05T16:06:55.589 DEBUG:teuthology.orchestra.run.smithi059:> kubectl create -f - 2022-08-05T16:06:56.513 INFO:teuthology.orchestra.run.smithi059.stdout:configmap/rook-config-override created 2022-08-05T16:06:56.540 INFO:tasks.rook:Cluster: apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: allowUnsupported: true image: quay.ceph.io/ceph-ci/ceph:4a7e6c4ac29de794744a2ce76ca2b36bc637894d dataDirHostPath: /var/lib/rook mgr: count: 1 modules: - enabled: true name: rook mon: allowMultiplePerNode: true count: 3 skipUpgradeChecks: true storage: storageClassDeviceSets: - count: 12 name: scratch portable: false volumeClaimTemplates: - metadata: name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: scratch volumeMode: Block 2022-08-05T16:06:56.541 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-08-05T16:06:56.542 DEBUG:teuthology.orchestra.run.smithi059:> dd of=cluster.yaml 2022-08-05T16:06:56.601 DEBUG:teuthology.orchestra.run.smithi059:> kubectl create -f cluster.yaml 2022-08-05T16:06:58.759 INFO:teuthology.orchestra.run.smithi059.stdout:cephcluster.ceph.rook.io/rook-ceph created 2022-08-05T16:06:58.760 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.756527 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-08-05T16:06:58.788 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.767560 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-08-05T16:06:58.788 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.774484 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph" 2022-08-05T16:06:58.793 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.794000 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster. 2022-08-05T16:06:58.803 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.798715 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config" 2022-08-05T16:06:58.835 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.810527 I | ceph-spec: detecting the ceph image version for image quay.ceph.io/ceph-ci/ceph:4a7e6c4ac29de794744a2ce76ca2b36bc637894d... 2022-08-05T16:06:58.873 DEBUG:teuthology.orchestra.remote:smithi059:rook/deploy/examples/operator.yaml is 24KB 2022-08-05T16:06:58.937 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.883999 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap) 2022-08-05T16:06:59.013 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884023 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap) 2022-08-05T16:06:59.014 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884031 I | op-k8sutil: ROOK_CSI_ENABLE_NFS="false" (configmap) 2022-08-05T16:06:59.014 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884037 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap) 2022-08-05T16:06:59.015 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884043 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap) 2022-08-05T16:06:59.015 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884053 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-08-05T16:06:59.015 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884061 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap) 2022-08-05T16:06:59.016 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884067 I | op-k8sutil: CSI_GRPC_TIMEOUT_SECONDS="150" (configmap) 2022-08-05T16:06:59.016 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884076 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-08-05T16:06:59.016 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884086 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default) 2022-08-05T16:06:59.017 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884097 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-08-05T16:06:59.017 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884107 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default) 2022-08-05T16:06:59.018 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884120 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-08-05T16:06:59.143 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884130 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default) 2022-08-05T16:06:59.144 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884140 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-08-05T16:06:59.144 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884152 I | op-k8sutil: CSIADDONS_PORT="9070" (default) 2022-08-05T16:06:59.144 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884162 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-08-05T16:06:59.145 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884169 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default) 2022-08-05T16:06:59.145 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884175 I | op-k8sutil: CSI_ENABLE_LIVENESS="false" (configmap) 2022-08-05T16:06:59.145 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884182 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="system-node-critical" (configmap) 2022-08-05T16:06:59.149 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884189 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="system-cluster-critical" (configmap) 2022-08-05T16:06:59.149 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884195 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default) 2022-08-05T16:06:59.150 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884202 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap) 2022-08-05T16:06:59.150 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884212 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap) 2022-08-05T16:06:59.151 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884221 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (configmap) 2022-08-05T16:06:59.151 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884234 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap) 2022-08-05T16:06:59.152 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884243 I | op-k8sutil: CSI_ENABLE_ENCRYPTION="false" (configmap) 2022-08-05T16:06:59.152 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884254 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-08-05T16:06:59.153 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884265 I | op-k8sutil: CSI_NFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-08-05T16:06:59.153 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884275 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default) 2022-08-05T16:06:59.161 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884288 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap) 2022-08-05T16:06:59.171 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884299 I | ceph-csi: Kubernetes version is 1.24 2022-08-05T16:06:59.196 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884312 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="registry.k8s.io/sig-storage/csi-resizer:v1.4.0" (default) 2022-08-05T16:06:59.197 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884323 I | op-k8sutil: CSI_LOG_LEVEL="" (default) 2022-08-05T16:06:59.198 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.884338 I | op-k8sutil: CSI_SIDECAR_LOG_LEVEL="" (default) 2022-08-05T16:06:59.204 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.915047 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap) 2022-08-05T16:06:59.204 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.915072 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.6.2" (default) 2022-08-05T16:06:59.205 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.915080 I | op-k8sutil: ROOK_CSI_NFS_IMAGE="registry.k8s.io/sig-storage/nfsplugin:v4.0.0" (default) 2022-08-05T16:06:59.205 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.915088 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1" (default) 2022-08-05T16:06:59.207 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.915095 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="registry.k8s.io/sig-storage/csi-provisioner:v3.1.0" (default) 2022-08-05T16:06:59.207 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.915101 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="registry.k8s.io/sig-storage/csi-attacher:v3.4.0" (default) 2022-08-05T16:06:59.208 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.915110 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="registry.k8s.io/sig-storage/csi-snapshotter:v6.0.1" (default) 2022-08-05T16:06:59.209 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.915119 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default) 2022-08-05T16:06:59.209 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.915129 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.3.0" (default) 2022-08-05T16:06:59.210 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.915140 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.4.0" (default) 2022-08-05T16:06:59.211 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.915148 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default) 2022-08-05T16:06:59.211 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.915157 I | op-k8sutil: ROOK_CSI_NFS_POD_LABELS="" (default) 2022-08-05T16:06:59.212 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.915164 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default) 2022-08-05T16:06:59.213 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.915174 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.6.2" 2022-08-05T16:06:59.213 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.915292 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-08-05T16:06:59.214 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:06:58.915318 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-08-05T16:06:59.364 DEBUG:teuthology.orchestra.run.smithi059:> kubectl create -f rook/deploy/examples/toolbox.yaml 2022-08-05T16:06:59.713 INFO:teuthology.orchestra.run.smithi059.stdout:deployment.apps/rook-ceph-tools created 2022-08-05T16:06:59.714 INFO:tasks.rook:Waiting for tools container to start 2022-08-05T16:06:59.714 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:06:59.850 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:06:59.850 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 Pending 0 0s 2022-08-05T16:06:59.851 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:06:59.948 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:06:59.969 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-csi-detect-version-7flln 0/1 Init:0/1 0 0s 2022-08-05T16:06:59.971 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-detect-version-m5zjq 0/1 Init:0/1 0 1s 2022-08-05T16:06:59.971 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 45s 2022-08-05T16:06:59.971 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 0s 2022-08-05T16:07:04.980 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:07:05.081 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:07:05.083 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 6s 2022-08-05T16:07:05.127 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:07:05.307 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:07:05.308 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-csi-detect-version-7flln 0/1 Init:0/1 0 6s 2022-08-05T16:07:05.308 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-detect-version-m5zjq 0/1 Init:0/1 0 7s 2022-08-05T16:07:05.309 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 51s 2022-08-05T16:07:05.309 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 6s 2022-08-05T16:07:10.427 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:07:10.606 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:07:10.607 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 11s 2022-08-05T16:07:10.608 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:07:10.763 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:07:10.764 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-csi-detect-version-7flln 0/1 Init:0/1 0 11s 2022-08-05T16:07:10.764 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-detect-version-m5zjq 0/1 Init:0/1 0 12s 2022-08-05T16:07:10.780 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 56s 2022-08-05T16:07:10.780 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 11s 2022-08-05T16:07:15.793 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:07:15.883 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:07:15.883 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 16s 2022-08-05T16:07:15.884 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:07:15.992 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:07:15.992 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-csi-detect-version-7flln 0/1 Init:0/1 0 16s 2022-08-05T16:07:15.992 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-detect-version-m5zjq 0/1 Init:0/1 0 17s 2022-08-05T16:07:15.992 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 61s 2022-08-05T16:07:15.993 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 16s 2022-08-05T16:07:20.995 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:07:21.077 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:07:21.082 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 22s 2022-08-05T16:07:21.083 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:07:21.223 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:07:21.233 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-csi-detect-version-7flln 0/1 Init:0/1 0 22s 2022-08-05T16:07:21.234 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-detect-version-m5zjq 0/1 Init:0/1 0 23s 2022-08-05T16:07:21.234 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 67s 2022-08-05T16:07:21.235 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 22s 2022-08-05T16:07:26.294 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:07:26.473 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:07:26.485 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 27s 2022-08-05T16:07:26.486 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:07:26.648 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:07:26.649 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-csi-detect-version-7flln 0/1 Init:0/1 0 27s 2022-08-05T16:07:26.649 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-detect-version-m5zjq 0/1 Init:0/1 0 28s 2022-08-05T16:07:26.668 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 72s 2022-08-05T16:07:26.668 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 27s 2022-08-05T16:07:31.678 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:07:31.766 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:07:31.777 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 32s 2022-08-05T16:07:31.778 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:07:31.873 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:07:31.874 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-csi-detect-version-7flln 0/1 Init:0/1 0 32s 2022-08-05T16:07:31.874 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-detect-version-m5zjq 0/1 Init:0/1 0 33s 2022-08-05T16:07:31.875 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 77s 2022-08-05T16:07:31.875 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 32s 2022-08-05T16:07:36.880 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:07:37.012 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:07:37.041 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 38s 2022-08-05T16:07:37.042 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:07:37.246 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:07:37.247 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-csi-detect-version-7flln 0/1 Init:0/1 0 38s 2022-08-05T16:07:37.247 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-detect-version-m5zjq 0/1 Init:0/1 0 39s 2022-08-05T16:07:37.248 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 83s 2022-08-05T16:07:37.248 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 38s 2022-08-05T16:07:42.265 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:07:42.382 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:07:42.383 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 43s 2022-08-05T16:07:42.401 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:07:42.494 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:07:42.495 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-csi-detect-version-7flln 0/1 PodInitializing 0 43s 2022-08-05T16:07:42.498 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-detect-version-m5zjq 0/1 PodInitializing 0 44s 2022-08-05T16:07:42.499 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 88s 2022-08-05T16:07:42.507 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 43s 2022-08-05T16:07:47.528 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:07:47.637 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:07:47.638 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 48s 2022-08-05T16:07:47.639 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:07:47.765 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:07:47.766 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-csi-detect-version-7flln 0/1 PodInitializing 0 48s 2022-08-05T16:07:47.767 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-detect-version-m5zjq 0/1 PodInitializing 0 49s 2022-08-05T16:07:47.767 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 93s 2022-08-05T16:07:47.768 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 48s 2022-08-05T16:07:52.776 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:07:52.852 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:07:52.853 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 53s 2022-08-05T16:07:52.856 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:07:52.944 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:07:52.945 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-csi-detect-version-7flln 0/1 PodInitializing 0 53s 2022-08-05T16:07:52.945 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-detect-version-m5zjq 0/1 PodInitializing 0 54s 2022-08-05T16:07:52.951 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 98s 2022-08-05T16:07:52.953 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 53s 2022-08-05T16:07:57.965 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:07:58.073 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:07:58.074 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 59s 2022-08-05T16:07:58.076 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:07:58.183 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:07:58.184 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-csi-detect-version-7flln 0/1 PodInitializing 0 59s 2022-08-05T16:07:58.184 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-detect-version-m5zjq 0/1 PodInitializing 0 60s 2022-08-05T16:07:58.186 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 104s 2022-08-05T16:07:58.187 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 59s 2022-08-05T16:08:03.196 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:08:03.297 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:08:03.310 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 64s 2022-08-05T16:08:03.311 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:08:03.436 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:08:03.437 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-csi-detect-version-7flln 0/1 PodInitializing 0 64s 2022-08-05T16:08:03.437 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-detect-version-m5zjq 0/1 PodInitializing 0 65s 2022-08-05T16:08:03.451 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 109s 2022-08-05T16:08:03.451 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 64s 2022-08-05T16:08:08.457 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:08:08.631 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:08:08.632 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 69s 2022-08-05T16:08:08.664 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:08:08.819 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:08:08.820 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-csi-detect-version-7flln 0/1 PodInitializing 0 69s 2022-08-05T16:08:08.843 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-detect-version-m5zjq 0/1 PodInitializing 0 70s 2022-08-05T16:08:08.844 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 114s 2022-08-05T16:08:08.847 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 69s 2022-08-05T16:08:13.850 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:08:14.054 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:08:14.055 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 74s 2022-08-05T16:08:14.055 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:08:14.136 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:08:14.136 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-csi-detect-version-7flln 0/1 PodInitializing 0 75s 2022-08-05T16:08:14.137 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-detect-version-m5zjq 0/1 PodInitializing 0 76s 2022-08-05T16:08:14.137 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 2m 2022-08-05T16:08:14.137 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 75s 2022-08-05T16:08:19.147 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:08:19.235 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:08:19.236 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 80s 2022-08-05T16:08:19.236 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:08:19.404 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:08:19.404 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-csi-detect-version-7flln 0/1 PodInitializing 0 80s 2022-08-05T16:08:19.404 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-detect-version-m5zjq 0/1 PodInitializing 0 81s 2022-08-05T16:08:19.405 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 2m5s 2022-08-05T16:08:19.405 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 80s 2022-08-05T16:08:24.417 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:08:24.515 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:08:24.516 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 85s 2022-08-05T16:08:24.523 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:08:24.713 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:08:24.713 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-csi-detect-version-7flln 0/1 PodInitializing 0 85s 2022-08-05T16:08:24.751 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-detect-version-m5zjq 0/1 PodInitializing 0 86s 2022-08-05T16:08:24.752 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 2m10s 2022-08-05T16:08:24.752 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 85s 2022-08-05T16:08:26.322 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:08:26.320088 I | ceph-csi: Detected ceph CSI image version: "v3.6.2" 2022-08-05T16:08:26.333 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:08:26.329182 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default) 2022-08-05T16:08:26.333 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:08:26.329206 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default) 2022-08-05T16:08:26.333 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:08:26.329211 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default) 2022-08-05T16:08:26.340 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:08:26.329216 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default) 2022-08-05T16:08:26.341 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:08:26.329220 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default) 2022-08-05T16:08:26.341 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:08:26.329224 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default) 2022-08-05T16:08:26.350 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:08:26.329228 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default) 2022-08-05T16:08:26.409 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:08:26.402271 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default) 2022-08-05T16:08:26.410 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:08:26.402300 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default) 2022-08-05T16:08:26.411 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:08:26.402309 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default) 2022-08-05T16:08:26.494 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:08:26.478235 I | ceph-csi: successfully started CSI Ceph RBD driver 2022-08-05T16:08:26.498 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:08:26.478274 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default) 2022-08-05T16:08:26.499 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:08:26.478282 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default) 2022-08-05T16:08:26.524 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:08:26.478289 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default) 2022-08-05T16:08:26.601 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:08:26.597588 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default) 2022-08-05T16:08:26.601 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:08:26.597622 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default) 2022-08-05T16:08:26.601 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:08:26.597636 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default) 2022-08-05T16:08:26.753 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:08:26.675563 I | ceph-csi: successfully started CSI CephFS driver 2022-08-05T16:08:26.762 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:08:26.675598 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-08-05T16:08:26.794 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:08:26.782619 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com" 2022-08-05T16:08:26.795 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:08:26.782647 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap) 2022-08-05T16:08:26.916 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:08:26.910259 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com" 2022-08-05T16:08:26.916 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:08:26.910292 I | ceph-csi: CSI NFS driver disabled 2022-08-05T16:08:26.916 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:08:26.910303 I | op-k8sutil: removing daemonset csi-nfsplugin if it exists 2022-08-05T16:08:26.917 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:08:26.913172 I | op-k8sutil: removing deployment csi-nfsplugin-provisioner if it exists 2022-08-05T16:08:26.958 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:08:26.941895 I | ceph-csi: successfully removed CSI NFS driver 2022-08-05T16:08:29.758 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:08:29.859 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:08:29.859 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 90s 2022-08-05T16:08:29.860 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:08:30.001 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:08:30.038 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-qm5sb 0/4 ContainerCreating 0 3s 2022-08-05T16:08:30.096 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-w6jbr 0/4 ContainerCreating 0 3s 2022-08-05T16:08:30.096 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-ps8nq 0/2 ContainerCreating 0 3s 2022-08-05T16:08:30.097 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-v2j9g 0/2 ContainerCreating 0 3s 2022-08-05T16:08:30.097 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-j9h5p 0/2 ContainerCreating 0 3s 2022-08-05T16:08:30.098 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-8lx4n 0/5 ContainerCreating 0 3s 2022-08-05T16:08:30.098 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-gwvf5 0/5 ContainerCreating 0 3s 2022-08-05T16:08:30.099 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-rldqv 0/2 ContainerCreating 0 3s 2022-08-05T16:08:30.099 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-csi-detect-version-7flln 0/1 Terminating 0 90s 2022-08-05T16:08:30.099 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-detect-version-m5zjq 0/1 PodInitializing 0 91s 2022-08-05T16:08:30.100 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 2m15s 2022-08-05T16:08:30.100 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 90s 2022-08-05T16:08:35.138 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:08:35.280 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:08:35.280 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 96s 2022-08-05T16:08:35.281 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:08:35.433 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:08:35.434 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-qm5sb 0/4 ContainerCreating 0 9s 2022-08-05T16:08:35.435 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-w6jbr 0/4 ContainerCreating 0 9s 2022-08-05T16:08:35.435 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-ps8nq 0/2 ContainerCreating 0 9s 2022-08-05T16:08:35.442 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-v2j9g 0/2 ContainerCreating 0 9s 2022-08-05T16:08:35.455 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-j9h5p 0/2 ContainerCreating 0 9s 2022-08-05T16:08:35.455 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-8lx4n 0/5 ContainerCreating 0 9s 2022-08-05T16:08:35.456 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-gwvf5 0/5 ContainerCreating 0 9s 2022-08-05T16:08:35.456 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-rldqv 0/2 ContainerCreating 0 9s 2022-08-05T16:08:35.457 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-detect-version-m5zjq 0/1 PodInitializing 0 97s 2022-08-05T16:08:35.457 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 2m21s 2022-08-05T16:08:35.458 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 96s 2022-08-05T16:08:40.478 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:08:40.605 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:08:40.605 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 101s 2022-08-05T16:08:40.621 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:08:40.834 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:08:40.847 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-qm5sb 0/4 ContainerCreating 0 14s 2022-08-05T16:08:40.889 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-w6jbr 0/4 ContainerCreating 0 14s 2022-08-05T16:08:40.890 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-ps8nq 0/2 ContainerCreating 0 14s 2022-08-05T16:08:40.890 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-v2j9g 0/2 ContainerCreating 0 14s 2022-08-05T16:08:40.891 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-j9h5p 0/2 ContainerCreating 0 14s 2022-08-05T16:08:40.930 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-8lx4n 0/5 ContainerCreating 0 14s 2022-08-05T16:08:40.931 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-gwvf5 0/5 ContainerCreating 0 14s 2022-08-05T16:08:40.932 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-rldqv 0/2 ContainerCreating 0 14s 2022-08-05T16:08:40.938 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-detect-version-m5zjq 0/1 PodInitializing 0 102s 2022-08-05T16:08:40.938 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 2m26s 2022-08-05T16:08:40.938 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 101s 2022-08-05T16:08:45.946 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:08:46.095 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:08:46.096 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 107s 2022-08-05T16:08:46.097 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:08:46.214 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:08:46.226 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-qm5sb 0/4 ContainerCreating 0 20s 2022-08-05T16:08:46.227 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-w6jbr 0/4 ContainerCreating 0 20s 2022-08-05T16:08:46.256 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-ps8nq 0/2 ContainerCreating 0 20s 2022-08-05T16:08:46.256 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-v2j9g 0/2 ContainerCreating 0 20s 2022-08-05T16:08:46.256 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-j9h5p 0/2 ContainerCreating 0 20s 2022-08-05T16:08:46.288 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-8lx4n 0/5 ContainerCreating 0 20s 2022-08-05T16:08:46.289 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-gwvf5 0/5 ContainerCreating 0 20s 2022-08-05T16:08:46.289 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-rldqv 0/2 ContainerCreating 0 20s 2022-08-05T16:08:46.308 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-detect-version-m5zjq 0/1 PodInitializing 0 108s 2022-08-05T16:08:46.308 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 2m32s 2022-08-05T16:08:46.357 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 107s 2022-08-05T16:08:51.359 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:08:51.443 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:08:51.444 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 112s 2022-08-05T16:08:51.444 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:08:51.590 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:08:51.591 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-qm5sb 0/4 ContainerCreating 0 25s 2022-08-05T16:08:51.591 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-w6jbr 0/4 ContainerCreating 0 25s 2022-08-05T16:08:51.591 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-ps8nq 0/2 ContainerCreating 0 25s 2022-08-05T16:08:51.592 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-v2j9g 0/2 ContainerCreating 0 25s 2022-08-05T16:08:51.592 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-j9h5p 0/2 ContainerCreating 0 25s 2022-08-05T16:08:51.592 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-8lx4n 0/5 ContainerCreating 0 25s 2022-08-05T16:08:51.592 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-gwvf5 0/5 ContainerCreating 0 25s 2022-08-05T16:08:51.593 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-rldqv 0/2 ContainerCreating 0 25s 2022-08-05T16:08:51.593 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-detect-version-m5zjq 0/1 PodInitializing 0 113s 2022-08-05T16:08:51.593 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 2m37s 2022-08-05T16:08:51.593 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 112s 2022-08-05T16:08:56.681 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:08:56.844 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:08:56.844 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 117s 2022-08-05T16:08:56.845 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:08:57.003 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:08:57.095 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-qm5sb 0/4 ContainerCreating 0 30s 2022-08-05T16:08:57.095 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-w6jbr 0/4 ContainerCreating 0 30s 2022-08-05T16:08:57.096 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-ps8nq 0/2 ContainerCreating 0 30s 2022-08-05T16:08:57.106 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-v2j9g 0/2 ContainerCreating 0 30s 2022-08-05T16:08:57.106 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-j9h5p 0/2 ContainerCreating 0 30s 2022-08-05T16:08:57.107 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-8lx4n 0/5 ContainerCreating 0 30s 2022-08-05T16:08:57.110 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-gwvf5 0/5 ContainerCreating 0 30s 2022-08-05T16:08:57.111 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-rldqv 0/2 ContainerCreating 0 30s 2022-08-05T16:08:57.111 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-detect-version-m5zjq 0/1 PodInitializing 0 118s 2022-08-05T16:08:57.112 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 2m42s 2022-08-05T16:08:57.112 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 117s 2022-08-05T16:09:02.115 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:09:02.249 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:09:02.249 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 2m3s 2022-08-05T16:09:02.270 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:09:02.406 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:09:02.407 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-qm5sb 0/4 ContainerCreating 0 36s 2022-08-05T16:09:02.407 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-w6jbr 0/4 ContainerCreating 0 36s 2022-08-05T16:09:02.407 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-ps8nq 0/2 ContainerCreating 0 36s 2022-08-05T16:09:02.407 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-v2j9g 0/2 ContainerCreating 0 36s 2022-08-05T16:09:02.408 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-j9h5p 0/2 ContainerCreating 0 36s 2022-08-05T16:09:02.408 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-8lx4n 0/5 ContainerCreating 0 36s 2022-08-05T16:09:02.408 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-gwvf5 0/5 ContainerCreating 0 36s 2022-08-05T16:09:02.408 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-rldqv 0/2 ContainerCreating 0 36s 2022-08-05T16:09:02.408 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-detect-version-m5zjq 0/1 PodInitializing 0 2m4s 2022-08-05T16:09:02.409 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 2m48s 2022-08-05T16:09:02.409 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 2m3s 2022-08-05T16:09:06.276 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:06.232454 I | ceph-spec: detected ceph image version: "17.0.0-14045 quincy" 2022-08-05T16:09:06.277 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:06.232490 I | ceph-cluster-controller: validating ceph version from provided image 2022-08-05T16:09:06.298 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:06.243457 I | ceph-cluster-controller: cluster "rook-ceph": version "17.0.0-14045 quincy" detected for image "quay.ceph.io/ceph-ci/ceph:4a7e6c4ac29de794744a2ce76ca2b36bc637894d" 2022-08-05T16:09:06.407 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:06.392443 I | op-mon: start running mons 2022-08-05T16:09:06.505 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:06.483640 I | ceph-spec: creating mon secrets for a new cluster 2022-08-05T16:09:06.506 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:06.501589 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found 2022-08-05T16:09:06.506 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:06.504468 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[],"namespace":""}] data: mapping:{"node":{}} maxMonId:-1] 2022-08-05T16:09:07.211 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:07.206876 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-08-05T16:09:07.212 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:07.207117 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-08-05T16:09:07.411 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:09:07.535 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:09:07.536 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 2m8s 2022-08-05T16:09:07.536 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:09:07.657 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:09:07.679 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-qm5sb 0/4 ContainerCreating 0 41s 2022-08-05T16:09:07.680 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-w6jbr 0/4 ContainerCreating 0 41s 2022-08-05T16:09:07.680 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-ps8nq 0/2 ContainerCreating 0 41s 2022-08-05T16:09:07.680 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-v2j9g 0/2 ContainerCreating 0 41s 2022-08-05T16:09:07.681 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-j9h5p 0/2 ContainerCreating 0 41s 2022-08-05T16:09:07.681 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-8lx4n 0/5 ContainerCreating 0 41s 2022-08-05T16:09:07.694 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-gwvf5 0/5 ContainerCreating 0 41s 2022-08-05T16:09:07.694 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-rldqv 0/2 ContainerCreating 0 41s 2022-08-05T16:09:07.694 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-detect-version-m5zjq 0/1 Terminating 0 2m9s 2022-08-05T16:09:07.695 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 2m53s 2022-08-05T16:09:07.695 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 2m8s 2022-08-05T16:09:08.842 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:08.808415 I | op-mon: targeting the mon count 3 2022-08-05T16:09:08.843 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:08.815535 I | op-mon: created canary deployment rook-ceph-mon-a-canary 2022-08-05T16:09:08.935 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:08.888950 I | op-mon: created canary deployment rook-ceph-mon-b-canary 2022-08-05T16:09:08.935 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:08.918103 I | op-mon: created canary deployment rook-ceph-mon-c-canary 2022-08-05T16:09:09.849 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:09.829610 I | op-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to smithi201 2022-08-05T16:09:09.850 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:09.829629 I | op-mon: mon a assigned to node smithi201 2022-08-05T16:09:10.019 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:10.011342 I | op-mon: canary monitor deployment rook-ceph-mon-b-canary scheduled to smithi157 2022-08-05T16:09:10.020 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:10.011361 I | op-mon: mon b assigned to node smithi157 2022-08-05T16:09:10.207 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:10.206056 I | op-mon: canary monitor deployment rook-ceph-mon-c-canary scheduled to smithi201 2022-08-05T16:09:10.207 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:10.206076 I | op-mon: mon c assigned to node smithi201 2022-08-05T16:09:10.273 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:10.212571 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary" 2022-08-05T16:09:10.294 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:10.217152 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-b-canary" 2022-08-05T16:09:10.318 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:10.272012 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-c-canary" 2022-08-05T16:09:10.358 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:10.357664 I | op-mon: creating mon a 2022-08-05T16:09:10.469 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:10.452228 I | ceph-spec: parsing mon endpoints: 2022-08-05T16:09:10.470 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:10.452257 W | ceph-spec: ignoring invalid monitor 2022-08-05T16:09:10.470 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:10.452302 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap) 2022-08-05T16:09:10.471 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:10.452320 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket" 2022-08-05T16:09:10.471 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:10.452988 I | op-bucket-prov: successfully reconciled bucket provisioner 2022-08-05T16:09:10.481 INFO:tasks.rook.operator.smithi059.stdout:I0805 16:09:10.453109 1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph.ceph.rook.io/bucket" 2022-08-05T16:09:10.738 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:10.719927 I | op-mon: mon "a" ip is 10.100.249.68 2022-08-05T16:09:11.239 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:11.208399 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.249.68:6789"],"namespace":""}] data:a=10.100.249.68:6789 mapping:{"node":{"a":{"Name":"smithi201","Hostname":"smithi201","Address":"172.21.15.201"},"b":{"Name":"smithi157","Hostname":"smithi157","Address":"172.21.15.157"},"c":{"Name":"smithi201","Hostname":"smithi201","Address":"172.21.15.201"}}} maxMonId:-1] 2022-08-05T16:09:11.250 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:11.208450 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-08-05T16:09:11.251 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:11.208525 I | op-mon: monitor endpoints changed, updating the bootstrap peer token 2022-08-05T16:09:12.165 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:12.133667 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-08-05T16:09:12.211 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:12.202024 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-08-05T16:09:12.238 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:12.223087 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster 2022-08-05T16:09:12.415 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:12.412673 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a" 2022-08-05T16:09:12.741 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:09:12.865 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:09:12.886 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 2m13s 2022-08-05T16:09:12.886 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:09:13.007 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:09:13.029 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-qm5sb 0/4 ContainerCreating 0 46s 2022-08-05T16:09:13.029 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-w6jbr 0/4 ContainerCreating 0 46s 2022-08-05T16:09:13.029 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-ps8nq 0/2 ContainerCreating 0 46s 2022-08-05T16:09:13.069 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-v2j9g 0/2 ContainerCreating 0 46s 2022-08-05T16:09:13.069 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-j9h5p 2/2 Running 0 46s 2022-08-05T16:09:13.069 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-8lx4n 0/5 ContainerCreating 0 46s 2022-08-05T16:09:13.086 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-gwvf5 0/5 ContainerCreating 0 46s 2022-08-05T16:09:13.087 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-rldqv 0/2 ContainerCreating 0 46s 2022-08-05T16:09:13.087 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-mon-a-8449f484c-vkzd4 0/1 Init:0/2 0 0s 2022-08-05T16:09:13.087 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-mon-a-canary-5f76c4596c-4rckw 1/1 Terminating 0 4s 2022-08-05T16:09:13.087 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-mon-b-canary-55ddc79975-wvkgs 0/1 Terminating 0 4s 2022-08-05T16:09:13.088 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-mon-c-canary-5cbdd458c-psrvk 0/1 Terminating 0 4s 2022-08-05T16:09:13.088 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 2m58s 2022-08-05T16:09:13.088 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 2m13s 2022-08-05T16:09:13.208 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:13.207511 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.249.68:6789"],"namespace":""}] data:a=10.100.249.68:6789 mapping:{"node":{"a":{"Name":"smithi201","Hostname":"smithi201","Address":"172.21.15.201"},"b":{"Name":"smithi157","Hostname":"smithi157","Address":"172.21.15.157"},"c":{"Name":"smithi201","Hostname":"smithi201","Address":"172.21.15.201"}}} maxMonId:0] 2022-08-05T16:09:13.209 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:13.207545 I | op-mon: waiting for mon quorum with [a] 2022-08-05T16:09:13.426 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:13.425643 I | op-mon: mons running: [a] 2022-08-05T16:09:18.121 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:09:18.195 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:09:18.196 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 2m19s 2022-08-05T16:09:18.196 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:09:18.333 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:09:18.333 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-qm5sb 0/4 ContainerCreating 0 52s 2022-08-05T16:09:18.333 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-w6jbr 0/4 ContainerCreating 0 52s 2022-08-05T16:09:18.378 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-ps8nq 0/2 ContainerCreating 0 52s 2022-08-05T16:09:18.378 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-v2j9g 2/2 Running 0 52s 2022-08-05T16:09:18.378 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-j9h5p 2/2 Running 0 52s 2022-08-05T16:09:18.379 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-8lx4n 0/5 ContainerCreating 0 52s 2022-08-05T16:09:18.413 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-gwvf5 0/5 ContainerCreating 0 52s 2022-08-05T16:09:18.414 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-rldqv 0/2 ContainerCreating 0 52s 2022-08-05T16:09:18.414 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-mon-a-8449f484c-vkzd4 0/1 Init:1/2 0 6s 2022-08-05T16:09:18.415 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-mon-a-canary-5f76c4596c-4rckw 1/1 Terminating 0 10s 2022-08-05T16:09:18.493 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-mon-b-canary-55ddc79975-wvkgs 0/1 Terminating 0 10s 2022-08-05T16:09:18.494 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-mon-c-canary-5cbdd458c-psrvk 1/1 Terminating 0 10s 2022-08-05T16:09:18.494 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 3m4s 2022-08-05T16:09:18.495 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 2m19s 2022-08-05T16:09:23.506 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:09:23.631 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:09:23.632 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 2m24s 2022-08-05T16:09:23.635 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:09:23.849 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:09:23.948 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-qm5sb 0/4 ContainerCreating 0 57s 2022-08-05T16:09:23.948 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-w6jbr 0/4 ContainerCreating 0 57s 2022-08-05T16:09:23.949 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-ps8nq 0/2 ContainerCreating 0 57s 2022-08-05T16:09:23.949 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-v2j9g 2/2 Running 0 57s 2022-08-05T16:09:23.949 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-j9h5p 2/2 Running 0 57s 2022-08-05T16:09:24.010 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-8lx4n 0/5 ContainerCreating 0 57s 2022-08-05T16:09:24.010 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-gwvf5 0/5 ContainerCreating 0 57s 2022-08-05T16:09:24.015 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-rldqv 0/2 ContainerCreating 0 57s 2022-08-05T16:09:24.016 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-mon-a-8449f484c-vkzd4 0/1 Running 0 11s 2022-08-05T16:09:24.022 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-mon-a-canary-5f76c4596c-4rckw 1/1 Terminating 0 15s 2022-08-05T16:09:24.022 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-mon-b-canary-55ddc79975-wvkgs 0/1 Terminating 0 15s 2022-08-05T16:09:24.023 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-mon-c-canary-5cbdd458c-psrvk 1/1 Terminating 0 15s 2022-08-05T16:09:24.023 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 3m9s 2022-08-05T16:09:24.023 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 0/1 ContainerCreating 0 2m24s 2022-08-05T16:09:29.032 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods -l app=rook-ceph-tools 2022-08-05T16:09:29.156 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:09:29.157 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 1/1 Running 0 2m30s 2022-08-05T16:09:29.172 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:09:29.309 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:09:29.310 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-qm5sb 0/4 ContainerCreating 0 63s 2022-08-05T16:09:29.311 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-w6jbr 0/4 ContainerCreating 0 63s 2022-08-05T16:09:29.311 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-ps8nq 0/2 ContainerCreating 0 63s 2022-08-05T16:09:29.312 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-v2j9g 2/2 Running 0 63s 2022-08-05T16:09:29.312 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-j9h5p 2/2 Running 0 63s 2022-08-05T16:09:29.312 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-8lx4n 0/5 ContainerCreating 0 63s 2022-08-05T16:09:29.313 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-gwvf5 0/5 ContainerCreating 0 63s 2022-08-05T16:09:29.313 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-rldqv 2/2 Running 0 63s 2022-08-05T16:09:29.314 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-mon-a-8449f484c-vkzd4 0/1 Running 0 17s 2022-08-05T16:09:29.314 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-mon-a-canary-5f76c4596c-4rckw 1/1 Terminating 0 21s 2022-08-05T16:09:29.315 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-mon-b-canary-55ddc79975-wvkgs 1/1 Terminating 0 21s 2022-08-05T16:09:29.315 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-mon-c-canary-5cbdd458c-psrvk 1/1 Terminating 0 21s 2022-08-05T16:09:29.316 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 3m15s 2022-08-05T16:09:29.316 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 1/1 Running 0 2m30s 2022-08-05T16:09:29.318 INFO:tasks.rook:Waiting for 12 OSDs 2022-08-05T16:09:29.319 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:09:33.565 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:33.560649 I | op-mon: mons running: [a] 2022-08-05T16:09:33.711 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:09:33.737 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":1,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-08-05T16:09:33.738 INFO:tasks.rook: have 0/12 OSDs 2022-08-05T16:09:33.800 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:33.793600 I | op-mon: Monitors in quorum: [a] 2022-08-05T16:09:33.800 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:33.793637 I | op-mon: mons created: 1 2022-08-05T16:09:34.029 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:34.020510 I | op-mon: waiting for mon quorum with [a] 2022-08-05T16:09:34.037 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:34.032509 I | op-mon: mons running: [a] 2022-08-05T16:09:34.395 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:34.393395 I | op-mon: Monitors in quorum: [a] 2022-08-05T16:09:34.395 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:34.393440 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-08-05T16:09:34.785 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:34.781732 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-08-05T16:09:34.785 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:34.781773 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-08-05T16:09:35.322 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:35.320174 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-08-05T16:09:35.326 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:35.320210 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-08-05T16:09:35.709 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:35.708621 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-08-05T16:09:35.709 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:35.708654 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-08-05T16:09:35.999 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:35.998484 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-08-05T16:09:35.999 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:35.998514 I | op-config: deleting "log file" option from the mon configuration database 2022-08-05T16:09:36.232 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:36.227174 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-08-05T16:09:36.233 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:36.227206 I | op-mon: creating mon b 2022-08-05T16:09:36.350 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:36.345126 I | op-mon: mon "a" ip is 10.100.249.68 2022-08-05T16:09:36.375 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:36.374879 I | op-mon: mon "b" ip is 10.99.3.123 2022-08-05T16:09:36.558 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:36.558074 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.249.68:6789","10.99.3.123:6789"],"namespace":""}] data:a=10.100.249.68:6789,b=10.99.3.123:6789 mapping:{"node":{"a":{"Name":"smithi201","Hostname":"smithi201","Address":"172.21.15.201"},"b":{"Name":"smithi157","Hostname":"smithi157","Address":"172.21.15.157"},"c":{"Name":"smithi201","Hostname":"smithi201","Address":"172.21.15.201"}}} maxMonId:0] 2022-08-05T16:09:36.619 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:36.617557 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-08-05T16:09:36.619 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:36.617792 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-08-05T16:09:36.875 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:36.869923 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-08-05T16:09:36.891 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:36.879292 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-08-05T16:09:36.891 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:36.879315 I | op-mon: waiting for mon quorum with [a b] 2022-08-05T16:09:37.241 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:37.235515 I | op-mon: mons running: [a b] 2022-08-05T16:09:37.471 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:37.471436 I | op-mon: Monitors in quorum: [a] 2022-08-05T16:09:37.519 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:37.515973 I | op-mon: updating maxMonID from 0 to 1 after committing mon "b" 2022-08-05T16:09:38.248 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:38.232542 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.249.68:6789","10.99.3.123:6789"],"namespace":""}] data:a=10.100.249.68:6789,b=10.99.3.123:6789 mapping:{"node":{"a":{"Name":"smithi201","Hostname":"smithi201","Address":"172.21.15.201"},"b":{"Name":"smithi157","Hostname":"smithi157","Address":"172.21.15.157"},"c":{"Name":"smithi201","Hostname":"smithi201","Address":"172.21.15.201"}}} maxMonId:1] 2022-08-05T16:09:38.249 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:38.232596 I | op-mon: waiting for mon quorum with [a b] 2022-08-05T16:09:38.649 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:38.637855 I | op-mon: mons running: [a b] 2022-08-05T16:09:38.871 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:38.871438 I | op-mon: Monitors in quorum: [a] 2022-08-05T16:09:38.872 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:38.871465 I | op-mon: mons created: 2 2022-08-05T16:09:39.118 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:39.116172 I | op-mon: waiting for mon quorum with [a b] 2022-08-05T16:09:39.171 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:39.166015 I | op-mon: mons running: [a b] 2022-08-05T16:09:39.387 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:39.383703 W | op-mon: failed to find initial monitor b in mon map 2022-08-05T16:09:39.387 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:39.383731 W | op-mon: monitor b is not in quorum list 2022-08-05T16:09:43.753 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:09:44.116 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:09:44.127 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":1,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-08-05T16:09:44.128 INFO:tasks.rook: have 0/12 OSDs 2022-08-05T16:09:44.427 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:44.408024 I | op-mon: mons running: [a b] 2022-08-05T16:09:44.691 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:44.648076 W | op-mon: failed to find initial monitor b in mon map 2022-08-05T16:09:44.691 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:44.648100 W | op-mon: monitor b is not in quorum list 2022-08-05T16:09:49.679 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:49.666492 I | op-mon: mons running: [a b] 2022-08-05T16:09:49.926 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:49.916072 W | op-mon: failed to find initial monitor b in mon map 2022-08-05T16:09:49.926 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:49.916088 W | op-mon: monitor b is not in quorum list 2022-08-05T16:09:54.129 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:09:54.515 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:09:54.559 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":1,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-08-05T16:09:54.560 INFO:tasks.rook: have 0/12 OSDs 2022-08-05T16:09:54.948 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:54.942058 I | op-mon: mons running: [a b] 2022-08-05T16:09:55.197 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:55.197503 W | op-mon: failed to find initial monitor b in mon map 2022-08-05T16:09:55.198 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:09:55.197531 W | op-mon: monitor b is not in quorum list 2022-08-05T16:10:00.225 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:00.224456 I | op-mon: mons running: [a b] 2022-08-05T16:10:00.470 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:00.466413 W | op-mon: failed to find initial monitor b in mon map 2022-08-05T16:10:00.472 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:00.466431 W | op-mon: monitor b is not in quorum list 2022-08-05T16:10:04.582 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:10:04.958 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:10:04.981 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":1,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-08-05T16:10:04.982 INFO:tasks.rook: have 0/12 OSDs 2022-08-05T16:10:05.506 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:05.490975 I | op-mon: mon b is not yet running 2022-08-05T16:10:05.506 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:05.491001 I | op-mon: mons running: [a] 2022-08-05T16:10:10.514 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:10.512881 I | op-mon: mon b is not yet running 2022-08-05T16:10:10.515 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:10.512908 I | op-mon: mons running: [a] 2022-08-05T16:10:15.006 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:10:15.408 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:10:15.485 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":1,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-08-05T16:10:15.486 INFO:tasks.rook: have 0/12 OSDs 2022-08-05T16:10:15.546 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:15.538874 I | op-mon: mon b is not yet running 2022-08-05T16:10:15.547 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:15.538894 I | op-mon: mons running: [a] 2022-08-05T16:10:20.567 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:20.561971 I | op-mon: mon b is not yet running 2022-08-05T16:10:20.568 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:20.562004 I | op-mon: mons running: [a] 2022-08-05T16:10:25.490 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:10:25.585 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:25.584123 I | op-mon: mon b is not yet running 2022-08-05T16:10:25.611 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:25.584155 I | op-mon: mons running: [a] 2022-08-05T16:10:25.915 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:10:25.916 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":1,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-08-05T16:10:25.917 INFO:tasks.rook: have 0/12 OSDs 2022-08-05T16:10:30.609 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:30.607016 I | op-mon: mons running: [a b] 2022-08-05T16:10:32.274 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:32.252040 I | op-mon: Monitors in quorum: [a b] 2022-08-05T16:10:32.274 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:32.252067 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-08-05T16:10:32.482 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:32.479402 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-08-05T16:10:32.483 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:32.479449 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-08-05T16:10:32.711 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:32.707602 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-08-05T16:10:32.712 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:32.707634 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-08-05T16:10:32.964 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:32.938597 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-08-05T16:10:32.964 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:32.938624 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-08-05T16:10:33.168 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:33.166815 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-08-05T16:10:33.169 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:33.166848 I | op-config: deleting "log file" option from the mon configuration database 2022-08-05T16:10:33.402 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:33.396750 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-08-05T16:10:33.402 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:33.396773 I | op-mon: creating mon c 2022-08-05T16:10:33.536 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:33.529540 I | op-mon: mon "a" ip is 10.100.249.68 2022-08-05T16:10:33.600 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:33.598287 I | op-mon: mon "b" ip is 10.99.3.123 2022-08-05T16:10:33.667 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:33.655259 I | op-mon: mon "c" ip is 10.104.187.48 2022-08-05T16:10:33.865 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:33.853099 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.249.68:6789","10.99.3.123:6789","10.104.187.48:6789"],"namespace":""}] data:a=10.100.249.68:6789,b=10.99.3.123:6789,c=10.104.187.48:6789 mapping:{"node":{"a":{"Name":"smithi201","Hostname":"smithi201","Address":"172.21.15.201"},"b":{"Name":"smithi157","Hostname":"smithi157","Address":"172.21.15.157"},"c":{"Name":"smithi201","Hostname":"smithi201","Address":"172.21.15.201"}}} maxMonId:1] 2022-08-05T16:10:34.402 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:34.402628 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config 2022-08-05T16:10:34.403 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:34.402924 I | cephclient: generated admin config in /var/lib/rook/rook-ceph 2022-08-05T16:10:34.832 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:34.817520 I | op-mon: 2 of 3 expected mon deployments exist. creating new deployment(s). 2022-08-05T16:10:34.841 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:34.837762 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed 2022-08-05T16:10:34.847 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:34.847561 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update 2022-08-05T16:10:34.856 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:34.852808 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed 2022-08-05T16:10:34.858 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:34.860015 I | op-k8sutil: deployment "rook-ceph-mon-b" did not change, nothing to update 2022-08-05T16:10:35.011 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:35.011017 I | op-mon: updating maxMonID from 1 to 2 after committing mon "c" 2022-08-05T16:10:35.830 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:35.811241 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.100.249.68:6789","10.99.3.123:6789","10.104.187.48:6789"],"namespace":""}] data:a=10.100.249.68:6789,b=10.99.3.123:6789,c=10.104.187.48:6789 mapping:{"node":{"a":{"Name":"smithi201","Hostname":"smithi201","Address":"172.21.15.201"},"b":{"Name":"smithi157","Hostname":"smithi157","Address":"172.21.15.157"},"c":{"Name":"smithi201","Hostname":"smithi201","Address":"172.21.15.201"}}} maxMonId:2] 2022-08-05T16:10:35.831 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:35.811276 I | op-mon: waiting for mon quorum with [a b c] 2022-08-05T16:10:35.928 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:10:36.304 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:10:36.305 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":1,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-08-05T16:10:36.319 INFO:tasks.rook: have 0/12 OSDs 2022-08-05T16:10:36.439 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:36.406797 I | op-mon: mon c is not yet running 2022-08-05T16:10:36.454 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:36.406828 I | op-mon: mons running: [a b] 2022-08-05T16:10:36.645 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:36.639442 I | op-mon: Monitors in quorum: [a b] 2022-08-05T16:10:36.645 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:36.639475 I | op-mon: mons created: 3 2022-08-05T16:10:36.866 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:36.867415 I | op-mon: waiting for mon quorum with [a b c] 2022-08-05T16:10:37.004 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:37.005089 I | op-mon: mon c is not yet running 2022-08-05T16:10:37.005 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:37.005121 I | op-mon: mons running: [a b] 2022-08-05T16:10:42.043 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:42.040556 I | op-mon: mons running: [a b c] 2022-08-05T16:10:46.322 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:10:46.668 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:46.669019 I | op-mon: Monitors in quorum: [a b c] 2022-08-05T16:10:46.668 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:46.669059 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database 2022-08-05T16:10:46.743 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:10:46.759 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":1,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-08-05T16:10:46.759 INFO:tasks.rook: have 0/12 OSDs 2022-08-05T16:10:46.902 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:46.891584 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database 2022-08-05T16:10:46.902 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:46.891615 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database 2022-08-05T16:10:47.122 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:47.114994 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database 2022-08-05T16:10:47.122 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:47.115025 I | op-config: setting "global"="mon allow pool size one"="true" option to the mon configuration database 2022-08-05T16:10:47.344 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:47.336338 I | op-config: successfully set "global"="mon allow pool size one"="true" option to the mon configuration database 2022-08-05T16:10:47.345 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:47.336371 I | op-config: setting "global"="log to file"="false" option to the mon configuration database 2022-08-05T16:10:47.632 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:47.633465 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database 2022-08-05T16:10:47.633 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:47.633493 I | op-config: deleting "log file" option from the mon configuration database 2022-08-05T16:10:47.859 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:47.860204 I | op-config: successfully deleted "log file" option from the mon configuration database 2022-08-05T16:10:47.859 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:47.860242 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner" 2022-08-05T16:10:48.274 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:48.251609 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node" 2022-08-05T16:10:48.645 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:48.643480 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner" 2022-08-05T16:10:48.985 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:48.985255 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node" 2022-08-05T16:10:49.408 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:49.405048 I | ceph-csi: created kubernetes csi secrets for cluster "rook-ceph" 2022-08-05T16:10:49.412 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:49.405085 I | cephclient: getting or creating ceph auth key "client.crash" 2022-08-05T16:10:49.779 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:49.779088 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "rook-ceph" 2022-08-05T16:10:49.779 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:49.779127 I | cephclient: create rbd-mirror bootstrap peer token "client.rbd-mirror-peer" 2022-08-05T16:10:49.780 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:49.779136 I | cephclient: getting or creating ceph auth key "client.rbd-mirror-peer" 2022-08-05T16:10:50.169 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:50.167035 I | cephclient: successfully created rbd-mirror bootstrap peer token for cluster "rook-ceph" 2022-08-05T16:10:50.203 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:50.194700 I | op-mgr: start running mgr 2022-08-05T16:10:50.204 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:50.194732 I | cephclient: getting or creating ceph auth key "mgr.a" 2022-08-05T16:10:50.689 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:50.678001 I | op-config: setting "mon"="auth_allow_insecure_global_id_reclaim"="false" option to the mon configuration database 2022-08-05T16:10:51.242 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:51.234322 I | op-config: successfully set "mon"="auth_allow_insecure_global_id_reclaim"="false" option to the mon configuration database 2022-08-05T16:10:51.243 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:10:51.234352 I | op-config: insecure global ID is now disabled 2022-08-05T16:10:56.761 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:10:57.192 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:10:57.217 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":1,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-08-05T16:10:57.218 INFO:tasks.rook: have 0/12 OSDs 2022-08-05T16:11:07.220 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:11:07.748 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:11:07.788 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":1,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-08-05T16:11:07.789 INFO:tasks.rook: have 0/12 OSDs 2022-08-05T16:11:12.408 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:12.358716 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a" 2022-08-05T16:11:12.409 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:12.380288 I | op-mgr: setting services to point to mgr "a" 2022-08-05T16:11:12.621 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:12.621806 I | op-mgr: no need to update service "rook-ceph-mgr" 2022-08-05T16:11:12.624 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:12.621969 I | op-mgr: successful modules: balancer 2022-08-05T16:11:12.685 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:12.683502 I | op-osd: start running osds in namespace "rook-ceph" 2022-08-05T16:11:12.698 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:12.683537 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s" 2022-08-05T16:11:12.700 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:12.688197 I | op-osd: start provisioning the OSDs on PVCs, if needed 2022-08-05T16:11:12.700 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:12.690593 I | op-osd: creating 12 new PVCs for device set "scratch" 2022-08-05T16:11:12.701 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:12.693612 I | op-osd: successfully provisioned PVC "scratch-data-0pcxbg" 2022-08-05T16:11:12.755 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:12.703083 I | op-osd: successfully provisioned PVC "scratch-data-1kdwj5" 2022-08-05T16:11:12.757 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:12.730166 I | op-osd: successfully provisioned PVC "scratch-data-25s8pz" 2022-08-05T16:11:12.757 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:12.736074 I | op-osd: successfully provisioned PVC "scratch-data-3qhvzh" 2022-08-05T16:11:12.763 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:12.762860 I | op-osd: successfully provisioned PVC "scratch-data-448h5h" 2022-08-05T16:11:12.814 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:12.769289 I | op-osd: successfully provisioned PVC "scratch-data-5s9q62" 2022-08-05T16:11:12.856 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:12.857952 I | op-osd: successfully provisioned PVC "scratch-data-6wz964" 2022-08-05T16:11:12.871 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:12.869626 I | op-osd: successfully provisioned PVC "scratch-data-7gg5h6" 2022-08-05T16:11:12.987 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:12.985875 I | op-osd: successfully provisioned PVC "scratch-data-8xjvrj" 2022-08-05T16:11:13.226 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:13.186738 I | op-osd: successfully provisioned PVC "scratch-data-95rk59" 2022-08-05T16:11:13.397 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:13.386648 I | op-osd: successfully provisioned PVC "scratch-data-104pld4" 2022-08-05T16:11:13.584 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:13.585765 I | op-osd: successfully provisioned PVC "scratch-data-11f27rj" 2022-08-05T16:11:13.656 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:13.596506 I | op-osd: OSD will have its main bluestore block on "scratch-data-0pcxbg" 2022-08-05T16:11:14.044 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:14.022221 I | op-osd: started OSD provisioning job for PVC "scratch-data-0pcxbg" 2022-08-05T16:11:14.044 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:14.022258 I | op-osd: OSD will have its main bluestore block on "scratch-data-1kdwj5" 2022-08-05T16:11:14.411 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:14.412991 I | op-mgr: successful modules: prometheus 2022-08-05T16:11:14.412 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:14.414986 I | op-mgr: successful modules: dashboard 2022-08-05T16:11:14.417 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:14.416181 I | op-mgr: successful modules: mgr module(s) from the spec 2022-08-05T16:11:14.453 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:14.445884 I | op-osd: started OSD provisioning job for PVC "scratch-data-1kdwj5" 2022-08-05T16:11:14.453 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:14.445910 I | op-osd: OSD will have its main bluestore block on "scratch-data-25s8pz" 2022-08-05T16:11:14.818 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:14.803960 I | op-osd: started OSD provisioning job for PVC "scratch-data-25s8pz" 2022-08-05T16:11:14.819 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:14.803985 I | op-osd: OSD will have its main bluestore block on "scratch-data-3qhvzh" 2022-08-05T16:11:15.219 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:15.212727 I | op-osd: started OSD provisioning job for PVC "scratch-data-3qhvzh" 2022-08-05T16:11:15.220 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:15.212749 I | op-osd: OSD will have its main bluestore block on "scratch-data-448h5h" 2022-08-05T16:11:15.656 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:15.641615 I | op-osd: started OSD provisioning job for PVC "scratch-data-448h5h" 2022-08-05T16:11:15.656 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:15.641649 I | op-osd: OSD will have its main bluestore block on "scratch-data-5s9q62" 2022-08-05T16:11:16.003 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:16.002939 I | op-osd: started OSD provisioning job for PVC "scratch-data-5s9q62" 2022-08-05T16:11:16.003 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:16.002977 I | op-osd: OSD will have its main bluestore block on "scratch-data-6wz964" 2022-08-05T16:11:16.446 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:16.431161 I | op-osd: started OSD provisioning job for PVC "scratch-data-6wz964" 2022-08-05T16:11:16.446 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:16.431195 I | op-osd: OSD will have its main bluestore block on "scratch-data-7gg5h6" 2022-08-05T16:11:16.882 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:16.857527 I | op-osd: started OSD provisioning job for PVC "scratch-data-7gg5h6" 2022-08-05T16:11:16.883 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:16.857543 I | op-osd: OSD will have its main bluestore block on "scratch-data-8xjvrj" 2022-08-05T16:11:17.227 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:17.217400 I | op-osd: started OSD provisioning job for PVC "scratch-data-8xjvrj" 2022-08-05T16:11:17.228 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:17.217419 I | op-osd: OSD will have its main bluestore block on "scratch-data-95rk59" 2022-08-05T16:11:17.662 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:17.662670 I | op-osd: started OSD provisioning job for PVC "scratch-data-95rk59" 2022-08-05T16:11:17.662 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:17.662697 I | op-osd: OSD will have its main bluestore block on "scratch-data-104pld4" 2022-08-05T16:11:17.794 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:11:18.024 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:18.012207 I | op-osd: started OSD provisioning job for PVC "scratch-data-104pld4" 2022-08-05T16:11:18.024 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:18.012223 I | op-osd: OSD will have its main bluestore block on "scratch-data-11f27rj" 2022-08-05T16:11:18.275 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:11:18.297 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":1,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-08-05T16:11:18.318 INFO:tasks.rook: have 0/12 OSDs 2022-08-05T16:11:18.465 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:18.464335 I | op-osd: started OSD provisioning job for PVC "scratch-data-11f27rj" 2022-08-05T16:11:18.471 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:18.464383 I | op-osd: start provisioning the OSDs on nodes, if needed 2022-08-05T16:11:18.472 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:18.464394 I | op-osd: no nodes are defined for configuring OSDs on raw devices 2022-08-05T16:11:18.600 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:18.589894 I | op-osd: OSD orchestration status for PVC scratch-data-0pcxbg is "starting" 2022-08-05T16:11:18.601 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:18.589914 I | op-osd: OSD orchestration status for PVC scratch-data-104pld4 is "starting" 2022-08-05T16:11:18.601 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:18.589919 I | op-osd: OSD orchestration status for PVC scratch-data-11f27rj is "starting" 2022-08-05T16:11:18.601 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:18.589923 I | op-osd: OSD orchestration status for PVC scratch-data-1kdwj5 is "starting" 2022-08-05T16:11:18.601 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:18.589927 I | op-osd: OSD orchestration status for PVC scratch-data-25s8pz is "starting" 2022-08-05T16:11:18.602 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:18.589931 I | op-osd: OSD orchestration status for PVC scratch-data-3qhvzh is "starting" 2022-08-05T16:11:18.614 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:18.589936 I | op-osd: OSD orchestration status for PVC scratch-data-448h5h is "starting" 2022-08-05T16:11:18.614 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:18.589940 I | op-osd: OSD orchestration status for PVC scratch-data-5s9q62 is "starting" 2022-08-05T16:11:18.615 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:18.589944 I | op-osd: OSD orchestration status for PVC scratch-data-6wz964 is "starting" 2022-08-05T16:11:18.615 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:18.589948 I | op-osd: OSD orchestration status for PVC scratch-data-7gg5h6 is "starting" 2022-08-05T16:11:18.615 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:18.589952 I | op-osd: OSD orchestration status for PVC scratch-data-8xjvrj is "starting" 2022-08-05T16:11:18.636 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:18.589957 I | op-osd: OSD orchestration status for PVC scratch-data-95rk59 is "starting" 2022-08-05T16:11:28.367 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:11:29.463 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:29.456972 I | op-osd: OSD orchestration status for node scratch-data-0pcxbg is "orchestrating" 2022-08-05T16:11:29.605 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:11:29.644 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":2,"num_osds":0,"num_up_osds":0,"osd_up_since":0,"num_in_osds":0,"osd_in_since":0,"num_remapped_pgs":0} 2022-08-05T16:11:29.645 INFO:tasks.rook: have 0/12 OSDs 2022-08-05T16:11:30.323 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:30.319376 I | op-osd: OSD orchestration status for PVC scratch-data-0pcxbg is "orchestrating" 2022-08-05T16:11:30.581 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:30.441490 I | exec: timeout waiting for process ceph to return. Sending interrupt signal to the process 2022-08-05T16:11:30.593 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:30.457630 E | op-mgr: failed modules: "orchestrator modules". failed to set rook orchestrator backend: failed to set rook as the orchestrator backend: failed to complete command for set rook backend: timeout waiting for the command ceph to return 2022-08-05T16:11:31.642 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:31.631608 I | op-osd: OSD orchestration status for node scratch-data-448h5h is "orchestrating" 2022-08-05T16:11:32.382 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:32.365811 I | op-osd: OSD orchestration status for node scratch-data-1kdwj5 is "orchestrating" 2022-08-05T16:11:32.686 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:32.685310 I | op-osd: OSD orchestration status for PVC scratch-data-448h5h is "orchestrating" 2022-08-05T16:11:33.186 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:33.189127 I | op-osd: OSD orchestration status for node scratch-data-25s8pz is "orchestrating" 2022-08-05T16:11:33.549 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:33.378300 I | op-osd: OSD orchestration status for PVC scratch-data-1kdwj5 is "orchestrating" 2022-08-05T16:11:34.203 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:34.175444 I | op-osd: OSD orchestration status for PVC scratch-data-25s8pz is "orchestrating" 2022-08-05T16:11:34.443 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:34.441888 I | op-osd: OSD orchestration status for node scratch-data-3qhvzh is "orchestrating" 2022-08-05T16:11:35.234 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:35.232872 I | op-osd: OSD orchestration status for PVC scratch-data-3qhvzh is "orchestrating" 2022-08-05T16:11:35.360 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:35.347662 I | op-osd: OSD orchestration status for node scratch-data-6wz964 is "orchestrating" 2022-08-05T16:11:35.392 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:35.382484 I | op-osd: OSD orchestration status for node scratch-data-5s9q62 is "orchestrating" 2022-08-05T16:11:35.667 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:35.669133 I | op-osd: OSD orchestration status for node scratch-data-7gg5h6 is "orchestrating" 2022-08-05T16:11:36.171 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:36.152879 I | op-osd: OSD orchestration status for PVC scratch-data-6wz964 is "orchestrating" 2022-08-05T16:11:36.191 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:36.191436 I | op-osd: OSD orchestration status for PVC scratch-data-5s9q62 is "orchestrating" 2022-08-05T16:11:36.463 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:36.450803 I | op-osd: OSD orchestration status for PVC scratch-data-7gg5h6 is "orchestrating" 2022-08-05T16:11:39.675 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:11:39.995 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:39.993995 I | op-osd: OSD orchestration status for PVC scratch-data-0pcxbg is "completed" 2022-08-05T16:11:39.995 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:39.994030 I | op-osd: creating OSD 0 on PVC "scratch-data-0pcxbg" 2022-08-05T16:11:39.996 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:39.994044 I | op-osd: OSD will have its main bluestore block on "scratch-data-0pcxbg" 2022-08-05T16:11:40.199 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:11:40.324 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":10,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:11:40.325 INFO:tasks.rook: have 0/12 OSDs 2022-08-05T16:11:40.475 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:40.444331 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi157": the object has been modified; please apply your changes to the latest version and try again 2022-08-05T16:11:40.519 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:40.496720 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-smithi157": the object has been modified; please apply your changes to the latest version and try again 2022-08-05T16:11:40.793 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:40.761724 I | op-osd: OSD orchestration status for PVC scratch-data-448h5h is "completed" 2022-08-05T16:11:40.812 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:40.761747 I | op-osd: creating OSD 1 on PVC "scratch-data-448h5h" 2022-08-05T16:11:40.812 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:40.761760 I | op-osd: OSD will have its main bluestore block on "scratch-data-448h5h" 2022-08-05T16:11:41.201 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:41.194284 I | op-osd: OSD orchestration status for PVC scratch-data-1kdwj5 is "completed" 2022-08-05T16:11:41.201 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:41.194310 I | op-osd: creating OSD 2 on PVC "scratch-data-1kdwj5" 2022-08-05T16:11:41.201 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:41.194319 I | op-osd: OSD will have its main bluestore block on "scratch-data-1kdwj5" 2022-08-05T16:11:41.356 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:41.358234 I | op-osd: OSD orchestration status for PVC scratch-data-25s8pz is "completed" 2022-08-05T16:11:41.357 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:41.358283 I | op-osd: creating OSD 3 on PVC "scratch-data-25s8pz" 2022-08-05T16:11:41.357 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:41.358300 I | op-osd: OSD will have its main bluestore block on "scratch-data-25s8pz" 2022-08-05T16:11:43.187 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:43.189614 I | op-osd: OSD orchestration status for PVC scratch-data-3qhvzh is "completed" 2022-08-05T16:11:43.188 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:43.189651 I | op-osd: creating OSD 4 on PVC "scratch-data-3qhvzh" 2022-08-05T16:11:43.190 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:43.189670 I | op-osd: OSD will have its main bluestore block on "scratch-data-3qhvzh" 2022-08-05T16:11:43.860 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:43.832624 I | op-osd: OSD orchestration status for PVC scratch-data-6wz964 is "completed" 2022-08-05T16:11:43.860 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:43.832660 I | op-osd: creating OSD 5 on PVC "scratch-data-6wz964" 2022-08-05T16:11:43.861 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:43.832674 I | op-osd: OSD will have its main bluestore block on "scratch-data-6wz964" 2022-08-05T16:11:43.989 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:43.987795 I | op-osd: OSD orchestration status for PVC scratch-data-5s9q62 is "completed" 2022-08-05T16:11:43.989 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:43.987827 I | op-osd: creating OSD 6 on PVC "scratch-data-5s9q62" 2022-08-05T16:11:43.998 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:43.987838 I | op-osd: OSD will have its main bluestore block on "scratch-data-5s9q62" 2022-08-05T16:11:44.316 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:44.313072 I | op-osd: OSD orchestration status for PVC scratch-data-7gg5h6 is "completed" 2022-08-05T16:11:44.317 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:44.313107 I | op-osd: creating OSD 7 on PVC "scratch-data-7gg5h6" 2022-08-05T16:11:44.317 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:11:44.313123 I | op-osd: OSD will have its main bluestore block on "scratch-data-7gg5h6" 2022-08-05T16:11:50.339 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:11:51.482 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:11:51.507 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":10,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:11:51.508 INFO:tasks.rook: have 0/12 OSDs 2022-08-05T16:12:01.516 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:12:03.443 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:12:03.465 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":10,"num_osds":8,"num_up_osds":0,"osd_up_since":0,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:12:03.466 INFO:tasks.rook: have 0/12 OSDs 2022-08-05T16:12:13.478 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:12:14.075 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:12:14.093 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":13,"num_osds":8,"num_up_osds":1,"osd_up_since":1659715933,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:12:14.094 INFO:tasks.rook: have 1/12 OSDs 2022-08-05T16:12:18.470 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:12:18.465320 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-08-05T16:12:24.095 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:12:24.757 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:12:24.757 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":22,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:12:24.762 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:12:34.771 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:12:35.342 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:12:35.355 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:12:35.356 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:12:45.370 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:12:46.104 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:12:46.129 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:12:46.159 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:12:56.164 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:12:56.679 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:12:56.774 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:12:56.775 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:13:06.786 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:13:07.259 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:13:07.320 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:13:07.321 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:13:17.330 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:13:17.887 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:13:17.929 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:13:17.931 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:13:18.465 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:13:18.464565 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-08-05T16:13:27.933 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:13:28.584 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:13:28.664 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:13:28.665 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:13:38.667 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:13:39.179 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:13:39.332 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:13:39.333 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:13:49.401 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:13:49.964 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:13:49.999 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:13:50.026 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:14:00.060 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:14:00.622 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:14:00.691 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:14:00.692 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:14:10.702 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:14:11.218 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:14:11.237 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:14:11.237 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:14:18.477 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:14:18.465024 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-08-05T16:14:21.245 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:14:21.802 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:14:21.823 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:14:21.824 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:14:31.843 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:14:32.480 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:14:32.483 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:14:32.486 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:14:42.530 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:14:43.129 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:14:43.167 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:14:43.168 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:14:53.178 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:14:53.718 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:14:53.753 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:14:53.754 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:15:03.756 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:15:04.298 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:15:04.298 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:15:04.317 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:15:14.322 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:15:14.931 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:15:14.957 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:15:14.959 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:15:18.497 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:15:18.465252 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-08-05T16:15:24.970 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:15:25.541 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:15:25.561 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:15:25.562 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:15:35.564 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:15:36.157 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:15:36.178 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:15:36.180 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:15:46.183 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:15:46.761 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:15:46.785 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:15:46.786 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:15:56.795 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:15:57.352 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:15:57.387 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:15:57.388 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:16:07.390 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:16:07.901 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:16:07.935 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:16:07.942 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:16:17.945 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:16:18.469 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:16:18.464644 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-08-05T16:16:18.482 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:16:18.506 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:16:18.507 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:16:28.517 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:16:29.042 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:16:29.054 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:16:29.055 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:16:39.063 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:16:39.598 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:16:39.637 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:16:39.664 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:16:49.675 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:16:50.202 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:16:50.223 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:16:50.224 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:17:00.226 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:17:00.735 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:17:00.767 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:17:00.768 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:17:10.794 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:17:11.330 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:17:11.352 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:17:11.353 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:17:18.473 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:17:18.465496 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-08-05T16:17:21.355 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:17:21.851 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:17:21.874 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:17:21.874 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:17:31.877 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:17:32.411 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:17:32.432 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:17:32.433 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:17:42.444 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:17:43.039 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:17:43.040 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:17:43.073 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:17:53.078 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:17:53.609 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:17:53.632 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:17:53.633 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:18:03.634 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:18:04.134 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:18:04.170 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:18:04.171 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:18:14.183 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:18:14.772 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:18:14.799 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:18:14.799 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:18:18.467 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:18:18.465514 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-08-05T16:18:24.809 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:18:25.433 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:18:25.456 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:18:25.460 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:18:35.464 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:18:35.979 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:18:35.993 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:18:35.994 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:18:45.998 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:18:46.506 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:18:46.523 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:18:46.524 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:18:56.526 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:18:57.142 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:18:57.170 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:18:57.171 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:19:07.174 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:19:07.720 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:19:07.743 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:19:07.748 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:19:17.750 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:19:18.286 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:19:18.308 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:19:18.309 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:19:18.466 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:19:18.465175 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-08-05T16:19:28.310 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:19:28.871 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:19:28.894 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:19:28.895 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:19:38.897 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:19:39.410 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:19:39.433 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:19:39.434 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:19:49.435 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:19:49.956 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:19:49.978 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:19:49.979 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:19:59.981 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:20:00.497 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:20:00.513 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:20:00.514 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:20:10.515 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:20:11.020 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:20:11.043 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:20:11.044 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:20:18.466 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:20:18.465281 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-08-05T16:20:21.045 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:20:21.566 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:20:21.585 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:20:21.586 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:20:31.593 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:20:32.104 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:20:32.131 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:20:32.132 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:20:42.133 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:20:42.650 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:20:42.673 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:20:42.673 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:20:52.674 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:20:53.176 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:20:53.198 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:20:53.198 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:21:03.200 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:21:03.709 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:21:03.730 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:21:03.731 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:21:13.733 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:21:14.233 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:21:14.258 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:21:14.258 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:21:18.466 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:21:18.465250 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-08-05T16:21:24.261 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:21:24.785 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:21:24.806 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:21:24.806 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:21:34.809 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:21:35.330 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:21:35.353 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:21:35.353 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:21:45.358 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:21:45.866 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:21:45.885 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:21:45.885 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:21:55.892 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:21:56.413 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:21:56.432 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:21:56.435 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:22:06.439 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:22:07.000 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:22:07.029 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:22:07.030 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:22:17.032 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:22:17.611 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:22:17.632 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:22:17.633 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:22:18.466 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:22:18.465112 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-08-05T16:22:27.635 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:22:28.142 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:22:28.161 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:22:28.162 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:22:38.164 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:22:38.692 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:22:38.713 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:22:38.713 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:22:48.728 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:22:49.256 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:22:49.273 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:22:49.275 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:22:59.277 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:22:59.785 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:22:59.820 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:22:59.821 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:23:09.823 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:23:10.361 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:23:10.382 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:23:10.382 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:23:18.468 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:23:18.465002 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-08-05T16:23:20.390 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:23:20.916 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:23:20.946 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:23:20.947 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:23:30.949 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:23:31.479 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:23:31.524 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:23:31.525 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:23:41.533 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:23:42.098 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:23:42.136 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:23:42.137 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:23:52.140 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:23:52.663 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:23:52.674 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:23:52.720 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:24:02.724 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:24:03.321 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:24:03.352 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:24:03.370 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:24:13.375 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:24:13.914 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:24:13.949 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:24:13.977 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:24:18.465 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:24:18.464920 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-08-05T16:24:23.982 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:24:24.540 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:24:24.580 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:24:24.581 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:24:34.585 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:24:35.296 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:24:35.296 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:24:35.297 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:24:45.338 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:24:45.876 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:24:45.907 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:24:45.908 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:24:55.923 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:24:56.432 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:24:56.580 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:24:56.580 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:25:06.589 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:25:07.096 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:25:07.112 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:25:07.113 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:25:17.122 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph exec rook-ceph-tools-6db9f859bb-8zq5r -- ceph osd stat -f json 2022-08-05T16:25:17.689 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:25:17.698 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1659715943,"num_in_osds":8,"osd_in_since":1659715897,"num_remapped_pgs":0} 2022-08-05T16:25:17.699 INFO:tasks.rook: have 8/12 OSDs 2022-08-05T16:25:17.700 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_9e4bf63f00ab3f82a26635c2779f4c3f1b73fb53/teuthology/contextutil.py", line 31, in nested vars.append(enter()) File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_4a7e6c4ac29de794744a2ce76ca2b36bc637894d/qa/tasks/rook.py", line 491, in wait_for_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_9e4bf63f00ab3f82a26635c2779f4c3f1b73fb53/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'check osd count' reached maximum tries (90) after waiting for 900 seconds 2022-08-05T16:25:17.701 ERROR:tasks.rook:'check osd count' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_4a7e6c4ac29de794744a2ce76ca2b36bc637894d/qa/tasks/rook.py", line 471, in rook_toolbox yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_9e4bf63f00ab3f82a26635c2779f4c3f1b73fb53/teuthology/contextutil.py", line 31, in nested vars.append(enter()) File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_4a7e6c4ac29de794744a2ce76ca2b36bc637894d/qa/tasks/rook.py", line 491, in wait_for_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_9e4bf63f00ab3f82a26635c2779f4c3f1b73fb53/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'check osd count' reached maximum tries (90) after waiting for 900 seconds 2022-08-05T16:25:17.823 DEBUG:teuthology.orchestra.remote:smithi059:rook/deploy/examples/operator.yaml is 24KB 2022-08-05T16:25:18.283 DEBUG:teuthology.orchestra.run.smithi059:> kubectl delete -f rook/deploy/examples/toolbox.yaml 2022-08-05T16:25:18.413 INFO:teuthology.orchestra.run.smithi059.stdout:deployment.apps "rook-ceph-tools" deleted 2022-08-05T16:25:18.474 ERROR:tasks.rook:'check osd count' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_4a7e6c4ac29de794744a2ce76ca2b36bc637894d/qa/tasks/rook.py", line 403, in rook_cluster yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_9e4bf63f00ab3f82a26635c2779f4c3f1b73fb53/teuthology/contextutil.py", line 31, in nested vars.append(enter()) File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_4a7e6c4ac29de794744a2ce76ca2b36bc637894d/qa/tasks/rook.py", line 491, in wait_for_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_9e4bf63f00ab3f82a26635c2779f4c3f1b73fb53/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'check osd count' reached maximum tries (90) after waiting for 900 seconds 2022-08-05T16:25:18.512 DEBUG:teuthology.orchestra.run.smithi059:> kubectl delete -f cluster.yaml 2022-08-05T16:25:18.515 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.465153 I | op-osd: waiting... 8 of 12 OSD prepare jobs have finished processing and 0 of 0 OSDs have been updated 2022-08-05T16:25:18.608 INFO:teuthology.orchestra.run.smithi059.stdout:cephcluster.ceph.rook.io "rook-ceph" deleted 2022-08-05T16:25:18.611 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.583006 I | ceph-cluster-controller: CR "rook-ceph" is going be deleted, cancelling any ongoing orchestration 2022-08-05T16:25:18.612 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.583151 I | operator: reloading operator's CRDs manager, cancelling all orchestrations! 2022-08-05T16:25:18.612 INFO:tasks.rook.operator.smithi059.stdout:I0805 16:25:18.583289 1 manager.go:148] objectbucket.io/provisioner-manager "msg"="stopping provisioner" "name"="rook-ceph.ceph.rook.io/bucket" "reason"="context canceled" 2022-08-05T16:25:18.612 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.583331 I | op-osd: context cancelled, exiting OSD update and create loop 2022-08-05T16:25:18.612 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.609531 I | ceph-cluster-controller: context cancelled, exiting reconcile 2022-08-05T16:25:18.613 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.610075 I | operator: successfully started the controller-runtime manager 2022-08-05T16:25:18.643 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.616259 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var) 2022-08-05T16:25:18.643 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.616282 I | operator: watching all namespaces for Ceph CRs 2022-08-05T16:25:18.643 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.616324 I | operator: setting up schemes 2022-08-05T16:25:18.643 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.619089 I | operator: setting up the controller-runtime manager 2022-08-05T16:25:19.263 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.823264 I | operator: Fetching webhook cert-manager-webhook to see if cert-manager is installed. 2022-08-05T16:25:19.263 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.826151 I | operator: failed to get cert manager 2022-08-05T16:25:19.263 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.826189 I | ceph-cluster-controller: successfully started 2022-08-05T16:25:19.264 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.826277 I | ceph-cluster-controller: enabling hotplug orchestration 2022-08-05T16:25:19.264 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.826307 I | ceph-crashcollector-controller: successfully started 2022-08-05T16:25:19.264 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.826333 I | ceph-block-pool-controller: successfully started 2022-08-05T16:25:19.264 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.826393 I | ceph-object-store-user-controller: successfully started 2022-08-05T16:25:19.264 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.826456 I | ceph-object-realm-controller: successfully started 2022-08-05T16:25:19.265 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.826483 I | ceph-object-zonegroup-controller: successfully started 2022-08-05T16:25:19.265 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.826508 I | ceph-object-zone-controller: successfully started 2022-08-05T16:25:19.313 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.826740 I | ceph-object-controller: successfully started 2022-08-05T16:25:19.313 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.826808 I | ceph-file-controller: successfully started 2022-08-05T16:25:19.314 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.826872 I | ceph-nfs-controller: successfully started 2022-08-05T16:25:19.314 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.826952 I | ceph-rbd-mirror-controller: successfully started 2022-08-05T16:25:19.314 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.826996 I | ceph-client-controller: successfully started 2022-08-05T16:25:19.314 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.827030 I | ceph-filesystem-mirror-controller: successfully started 2022-08-05T16:25:19.315 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.827082 I | operator: rook-ceph-operator-config-controller successfully started 2022-08-05T16:25:19.324 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.827117 I | ceph-csi: rook-ceph-operator-csi-controller successfully started 2022-08-05T16:25:19.324 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.827147 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started 2022-08-05T16:25:19.324 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.827175 I | ceph-bucket-topic: successfully started 2022-08-05T16:25:19.325 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.827202 I | ceph-bucket-notification: successfully started 2022-08-05T16:25:19.325 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.827229 I | ceph-bucket-notification: successfully started 2022-08-05T16:25:19.325 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.827256 I | ceph-fs-subvolumegroup-controller: successfully started 2022-08-05T16:25:19.325 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.827287 I | blockpool-rados-namespace-controller: successfully started 2022-08-05T16:25:19.326 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.828729 I | operator: starting the controller-runtime manager 2022-08-05T16:25:19.326 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.931073 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap) 2022-08-05T16:25:19.327 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.931162 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap) 2022-08-05T16:25:19.327 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.931185 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap) 2022-08-05T16:25:19.327 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.933945 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default) 2022-08-05T16:25:19.328 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.935772 I | operator: rook-ceph-operator-config-controller done reconciling 2022-08-05T16:25:19.328 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:18.999965 I | ceph-cluster-controller: CephCluster "rook-ceph/rook-ceph" can be deleted safely. deleting CephCluster "rook-ceph/rook-ceph" 2022-08-05T16:25:19.345 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.053820 I | ceph-cluster-controller: cleaning up CephCluster "rook-ceph/rook-ceph" 2022-08-05T16:25:19.346 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.053858 I | ceph-cluster-controller: checking any PVC created by drivers "rook-ceph.cephfs.csi.ceph.com" and "rook-ceph.rbd.csi.ceph.com" with clusterID "rook-ceph" 2022-08-05T16:25:19.346 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.058088 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi059-lv-1" 2022-08-05T16:25:19.346 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.058107 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi059-lv-2" 2022-08-05T16:25:19.347 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.058114 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi059-lv-3" 2022-08-05T16:25:19.347 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.058119 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi059-lv-4" 2022-08-05T16:25:19.347 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.058124 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi157-lv-1" 2022-08-05T16:25:19.347 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.058130 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi157-lv-2" 2022-08-05T16:25:19.348 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.058134 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi157-lv-3" 2022-08-05T16:25:19.359 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.058139 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi157-lv-4" 2022-08-05T16:25:19.359 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.058145 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi201-lv-1" 2022-08-05T16:25:19.359 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.058151 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi201-lv-2" 2022-08-05T16:25:19.360 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.058156 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi201-lv-3" 2022-08-05T16:25:19.360 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.058162 E | ceph-cluster-controller: Spec.CSI is nil for PV "smithi201-lv-4" 2022-08-05T16:25:19.360 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.058167 I | ceph-cluster-controller: no volume attachments for cluster "rook-ceph" 2022-08-05T16:25:19.361 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.058223 I | ceph-spec: removing finalizer "cephcluster.ceph.rook.io" on "rook-ceph" 2022-08-05T16:25:19.361 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.069183 E | clusterdisruption-controller: cephcluster "rook-ceph/rook-ceph" seems to be deleted, not requeuing until triggered again 2022-08-05T16:25:19.362 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.069922 I | ceph-spec: removing finalizer "ceph.rook.io/disaster-protection" on "rook-ceph-mon" 2022-08-05T16:25:19.362 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.070461 I | ceph-csi: CSI Ceph RBD driver disabled 2022-08-05T16:25:19.362 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.070484 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists 2022-08-05T16:25:19.363 INFO:tasks.rook:Waiting for cluster to stop 2022-08-05T16:25:19.364 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:25:19.368 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.074528 I | ceph-spec: removing finalizer "ceph.rook.io/disaster-protection" on "rook-ceph-mon-endpoints" 2022-08-05T16:25:19.368 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.091838 I | op-k8sutil: Removed daemonset csi-rbdplugin 2022-08-05T16:25:19.368 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.115992 I | ceph-spec: object "rook-ceph-osd-0" matched on delete, reconciling 2022-08-05T16:25:19.369 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.139218 I | ceph-spec: object "rook-ceph-admin-keyring" matched on delete, reconciling 2022-08-05T16:25:19.369 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.140603 I | ceph-spec: object "rook-ceph-mgr" matched on delete, reconciling 2022-08-05T16:25:19.369 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.140841 I | op-k8sutil: "csi-rbdplugin" still found. waiting... 2022-08-05T16:25:19.369 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.141251 I | ceph-spec: object "rook-ceph-mon-c" matched on delete, reconciling 2022-08-05T16:25:19.370 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.141838 I | ceph-spec: object "rook-ceph-mons-keyring" matched on delete, reconciling 2022-08-05T16:25:19.370 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.150083 I | ceph-spec: object "rook-ceph-mon-b" matched on delete, reconciling 2022-08-05T16:25:19.370 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.233518 I | ceph-spec: object "cluster-peer-token-rook-ceph" matched on delete, reconciling 2022-08-05T16:25:19.370 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.267667 I | ceph-spec: object "rook-ceph-osd-2" matched on delete, reconciling 2022-08-05T16:25:19.451 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.377479 I | ceph-spec: object "rook-ceph-crashcollector-smithi201" matched on delete, reconciling 2022-08-05T16:25:19.451 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.384721 I | ceph-spec: object "rook-ceph-mgr-a" matched on delete, reconciling 2022-08-05T16:25:19.530 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:25:19.530 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-qm5sb 4/4 Running 0 16m 2022-08-05T16:25:19.531 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-w6jbr 4/4 Running 0 16m 2022-08-05T16:25:19.531 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-ps8nq 2/2 Running 0 16m 2022-08-05T16:25:19.531 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-v2j9g 2/2 Running 0 16m 2022-08-05T16:25:19.531 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-j9h5p 2/2 Running 0 16m 2022-08-05T16:25:19.532 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-8lx4n 5/5 Running 0 16m 2022-08-05T16:25:19.542 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-gwvf5 5/5 Running 0 16m 2022-08-05T16:25:19.543 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-rldqv 2/2 Running 0 16m 2022-08-05T16:25:19.543 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-crashcollector-smithi157-c5c8bc664-sjggc 1/1 Running 0 13m 2022-08-05T16:25:19.544 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-crashcollector-smithi201-74bdcbddb-6nxlm 1/1 Running 0 14m 2022-08-05T16:25:19.544 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-mgr-a-77df5b7f55-49npg 1/1 Running 0 14m 2022-08-05T16:25:19.544 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-mon-a-8449f484c-vkzd4 1/1 Running 0 16m 2022-08-05T16:25:19.545 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-mon-b-9666d564c-8b5sz 1/1 Running 0 15m 2022-08-05T16:25:19.545 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-mon-c-849f5f79d7-5lhv7 1/1 Running 0 14m 2022-08-05T16:25:19.545 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 19m 2022-08-05T16:25:19.545 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-0-5657dc9dcc-hds45 1/1 Running 0 13m 2022-08-05T16:25:19.574 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-1-745d44cff7-ct4md 1/1 Running 0 13m 2022-08-05T16:25:19.584 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-2-777667c84d-m2466 1/1 Running 0 13m 2022-08-05T16:25:19.584 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-3-56754b977f-fbwbc 1/1 Running 0 13m 2022-08-05T16:25:19.584 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-4-5766f77b6b-qphzr 1/1 Running 0 13m 2022-08-05T16:25:19.585 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-5-86b478696c-4kg72 1/1 Running 0 13m 2022-08-05T16:25:19.585 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-6-b6f549474-w5f4s 1/1 Running 0 13m 2022-08-05T16:25:19.585 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-7-5f5fb5fb4b-hvkx9 1/1 Running 0 13m 2022-08-05T16:25:19.586 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-prepare-scratch-data-0pcxbg-7f2kd 0/1 Completed 0 14m 2022-08-05T16:25:19.586 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-prepare-scratch-data-104pld4-m22nr 0/1 Pending 0 14m 2022-08-05T16:25:19.586 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-prepare-scratch-data-11f27rj-kdfbx 0/1 Pending 0 14m 2022-08-05T16:25:19.586 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-prepare-scratch-data-1kdwj5-6wxc4 0/1 Completed 0 14m 2022-08-05T16:25:19.587 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-prepare-scratch-data-25s8pz-vgfsx 0/1 Completed 0 14m 2022-08-05T16:25:19.587 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-prepare-scratch-data-3qhvzh-n5wvh 0/1 Completed 0 14m 2022-08-05T16:25:19.587 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-prepare-scratch-data-448h5h-8ftw7 0/1 Completed 0 14m 2022-08-05T16:25:19.587 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-prepare-scratch-data-5s9q62-2sx66 0/1 Completed 0 14m 2022-08-05T16:25:19.588 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-prepare-scratch-data-6wz964-64ttj 0/1 Completed 0 14m 2022-08-05T16:25:19.588 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-prepare-scratch-data-7gg5h6-2kptj 0/1 Completed 0 14m 2022-08-05T16:25:19.588 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-prepare-scratch-data-8xjvrj-4bkk7 0/1 Pending 0 14m 2022-08-05T16:25:19.588 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-prepare-scratch-data-95rk59-llztf 0/1 Pending 0 14m 2022-08-05T16:25:19.588 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 1/1 Terminating 0 18m 2022-08-05T16:25:19.774 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.773902 I | ceph-spec: object "rook-csi-cephfs-provisioner" matched on delete, reconciling 2022-08-05T16:25:19.827 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.807459 I | ceph-spec: object "rook-ceph-mon" matched on delete, reconciling 2022-08-05T16:25:19.899 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.898286 I | ceph-spec: object "rook-csi-cephfs-node" matched on delete, reconciling 2022-08-05T16:25:19.931 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:19.925920 I | ceph-spec: object "rook-ceph-osd-1" matched on delete, reconciling 2022-08-05T16:25:20.052 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:20.004126 I | ceph-spec: object "rook-ceph-mon-a" matched on delete, reconciling 2022-08-05T16:25:20.054 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:20.017685 I | ceph-spec: object "rook-ceph-crashcollector-smithi157" matched on delete, reconciling 2022-08-05T16:25:20.088 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:20.084671 I | ceph-spec: object "rook-ceph-osd-7" matched on delete, reconciling 2022-08-05T16:25:20.106 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:20.101559 I | ceph-spec: object "rook-csi-rbd-provisioner" matched on delete, reconciling 2022-08-05T16:25:20.266 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:20.243939 I | ceph-spec: object "rook-ceph-osd-3" matched on delete, reconciling 2022-08-05T16:25:20.297 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:20.291683 I | ceph-spec: object "rook-ceph-osd-5" matched on delete, reconciling 2022-08-05T16:25:20.388 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:20.369353 I | ceph-spec: object "rook-ceph-osd-6" matched on delete, reconciling 2022-08-05T16:25:20.775 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:20.772099 I | ceph-spec: object "rook-ceph-mon-endpoints" matched on delete, reconciling 2022-08-05T16:25:21.015 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:20.994998 I | ceph-spec: object "rook-csi-rbd-node" matched on delete, reconciling 2022-08-05T16:25:21.015 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:21.000632 I | ceph-spec: object "rook-ceph-mgr-a-keyring" matched on delete, reconciling 2022-08-05T16:25:21.167 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:21.162415 I | ceph-spec: object "rook-ceph-crash-collector-keyring" matched on delete, reconciling 2022-08-05T16:25:21.201 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:21.199178 I | ceph-spec: object "rook-ceph-osd-4" matched on delete, reconciling 2022-08-05T16:25:21.228 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:21.222746 I | ceph-spec: object "rook-ceph-config" matched on delete, reconciling 2022-08-05T16:25:24.595 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:25:24.788 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:25:24.788 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-qm5sb 4/4 Running 0 16m 2022-08-05T16:25:24.789 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-w6jbr 4/4 Running 0 16m 2022-08-05T16:25:24.789 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-ps8nq 2/2 Running 0 16m 2022-08-05T16:25:24.808 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-v2j9g 2/2 Running 0 16m 2022-08-05T16:25:24.809 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-j9h5p 2/2 Terminating 0 16m 2022-08-05T16:25:24.809 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-8lx4n 5/5 Running 0 16m 2022-08-05T16:25:24.809 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-gwvf5 5/5 Running 0 16m 2022-08-05T16:25:24.809 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-crashcollector-smithi157-c5c8bc664-sjggc 1/1 Terminating 0 13m 2022-08-05T16:25:24.814 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-crashcollector-smithi201-74bdcbddb-6nxlm 1/1 Terminating 0 14m 2022-08-05T16:25:24.814 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-mgr-a-77df5b7f55-49npg 1/1 Terminating 0 14m 2022-08-05T16:25:24.814 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-mon-a-8449f484c-vkzd4 1/1 Terminating 0 16m 2022-08-05T16:25:24.815 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-mon-b-9666d564c-8b5sz 1/1 Terminating 0 15m 2022-08-05T16:25:24.815 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-mon-c-849f5f79d7-5lhv7 1/1 Terminating 0 14m 2022-08-05T16:25:24.815 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 19m 2022-08-05T16:25:24.815 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-0-5657dc9dcc-hds45 1/1 Terminating 0 13m 2022-08-05T16:25:24.815 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-1-745d44cff7-ct4md 1/1 Terminating 0 13m 2022-08-05T16:25:24.816 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-2-777667c84d-m2466 1/1 Terminating 0 13m 2022-08-05T16:25:24.816 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-3-56754b977f-fbwbc 1/1 Terminating 0 13m 2022-08-05T16:25:24.816 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-4-5766f77b6b-qphzr 1/1 Terminating 0 13m 2022-08-05T16:25:24.816 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-5-86b478696c-4kg72 1/1 Terminating 0 13m 2022-08-05T16:25:24.816 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-6-b6f549474-w5f4s 1/1 Terminating 0 13m 2022-08-05T16:25:24.817 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-7-5f5fb5fb4b-hvkx9 1/1 Terminating 0 13m 2022-08-05T16:25:24.817 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 1/1 Terminating 0 18m 2022-08-05T16:25:25.259 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:25.246655 I | op-k8sutil: confirmed csi-rbdplugin does not exist 2022-08-05T16:25:25.259 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:25.246687 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists 2022-08-05T16:25:25.260 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:25.251752 I | op-k8sutil: Removed deployment csi-rbdplugin-provisioner 2022-08-05T16:25:25.301 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:25.284031 I | op-k8sutil: "csi-rbdplugin-provisioner" still found. waiting... 2022-08-05T16:25:29.827 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:25:29.960 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:25:29.961 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-qm5sb 4/4 Running 0 17m 2022-08-05T16:25:29.961 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-w6jbr 4/4 Running 0 17m 2022-08-05T16:25:29.961 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-ps8nq 2/2 Running 0 17m 2022-08-05T16:25:29.987 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-v2j9g 2/2 Running 0 17m 2022-08-05T16:25:29.987 INFO:teuthology.orchestra.run.smithi059.stdout:csi-rbdplugin-provisioner-5dcc5cb8b4-gwvf5 5/5 Terminating 0 17m 2022-08-05T16:25:29.988 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 19m 2022-08-05T16:25:29.988 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-0-5657dc9dcc-hds45 1/1 Terminating 0 13m 2022-08-05T16:25:29.988 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-1-745d44cff7-ct4md 1/1 Terminating 0 13m 2022-08-05T16:25:29.988 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-2-777667c84d-m2466 1/1 Terminating 0 13m 2022-08-05T16:25:29.989 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-3-56754b977f-fbwbc 1/1 Terminating 0 13m 2022-08-05T16:25:29.989 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-4-5766f77b6b-qphzr 1/1 Terminating 0 13m 2022-08-05T16:25:29.989 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-5-86b478696c-4kg72 1/1 Terminating 0 13m 2022-08-05T16:25:29.989 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-6-b6f549474-w5f4s 1/1 Terminating 0 13m 2022-08-05T16:25:30.009 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-osd-7-5f5fb5fb4b-hvkx9 1/1 Terminating 0 13m 2022-08-05T16:25:30.010 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 1/1 Terminating 0 18m 2022-08-05T16:25:31.336 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:31.335551 I | op-k8sutil: confirmed csi-rbdplugin-provisioner does not exist 2022-08-05T16:25:31.347 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:31.344603 I | ceph-csi: successfully removed CSI Ceph RBD driver 2022-08-05T16:25:31.348 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:31.344633 I | ceph-csi: CSI CephFS driver disabled 2022-08-05T16:25:31.348 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:31.344646 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists 2022-08-05T16:25:31.375 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:31.355422 I | op-k8sutil: Removed daemonset csi-cephfsplugin 2022-08-05T16:25:31.382 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:31.376132 I | op-k8sutil: "csi-cephfsplugin" still found. waiting... 2022-08-05T16:25:35.034 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph get pods 2022-08-05T16:25:35.133 INFO:teuthology.orchestra.run.smithi059.stdout:NAME READY STATUS RESTARTS AGE 2022-08-05T16:25:35.133 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-qm5sb 4/4 Running 0 17m 2022-08-05T16:25:35.165 INFO:teuthology.orchestra.run.smithi059.stdout:csi-cephfsplugin-provisioner-87d596b9f-w6jbr 4/4 Running 0 17m 2022-08-05T16:25:35.165 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-operator-b5c96c99b-5hsx5 1/1 Running 0 19m 2022-08-05T16:25:35.193 INFO:teuthology.orchestra.run.smithi059.stdout:rook-ceph-tools-6db9f859bb-8zq5r 1/1 Terminating 0 18m 2022-08-05T16:25:35.194 DEBUG:teuthology.orchestra.run.smithi059:> kubectl -n rook-ceph delete configmap rook-config-override 2022-08-05T16:25:35.271 INFO:teuthology.orchestra.run.smithi059.stdout:configmap "rook-config-override" deleted 2022-08-05T16:25:35.347 DEBUG:teuthology.orchestra.run.smithi059:> rm -f cluster.yaml 2022-08-05T16:25:35.409 INFO:tasks.rook:Checking cluster log for badness... 2022-08-05T16:25:35.411 DEBUG:teuthology.orchestra.run.smithi059:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/lib/rook/rook-ceph/log/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2022-08-05T16:25:35.527 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:35.443720 I | op-k8sutil: confirmed csi-cephfsplugin does not exist 2022-08-05T16:25:35.528 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:35.443755 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists 2022-08-05T16:25:35.528 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:35.450443 I | op-k8sutil: Removed deployment csi-cephfsplugin-provisioner 2022-08-05T16:25:35.603 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:35.487197 I | op-k8sutil: "csi-cephfsplugin-provisioner" still found. waiting... 2022-08-05T16:25:35.700 INFO:teuthology.orchestra.run.smithi059.stderr:grep: /var/lib/rook/rook-ceph/log/ceph.log: No such file or directory 2022-08-05T16:25:35.701 INFO:tasks.rook:Compressing logs... 2022-08-05T16:25:35.750 DEBUG:teuthology.orchestra.run.smithi059:> sudo find /var/lib/rook/rook-ceph/log -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-08-05T16:25:35.824 DEBUG:teuthology.orchestra.run.smithi157:> sudo find /var/lib/rook/rook-ceph/log -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-08-05T16:25:35.911 INFO:teuthology.orchestra.run.smithi059.stderr:find: '/var/lib/rook/rook-ceph/log': No such file or directory 2022-08-05T16:25:35.985 DEBUG:teuthology.orchestra.run.smithi201:> sudo find /var/lib/rook/rook-ceph/log -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-08-05T16:25:37.434 INFO:tasks.rook:Archiving logs... 2022-08-05T16:25:37.533 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/lib/rook/rook-ceph/log to /home/teuthworker/archive/yuriw-2022-08-05_14:49:51-rados-wip-yuri4-testing-2022-08-04-1413-distro-default-smithi/6959556/remote/ubuntu@smithi059.front.sepia.ceph.com/log 2022-08-05T16:25:37.540 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar cz -f - -C /var/lib/rook/rook-ceph/log -- . 2022-08-05T16:25:37.595 INFO:teuthology.orchestra.run.smithi059.stderr:tar: /var/lib/rook/rook-ceph/log: Cannot open: No such file or directory 2022-08-05T16:25:37.596 INFO:teuthology.orchestra.run.smithi059.stderr:tar: Error is not recoverable: exiting now 2022-08-05T16:25:37.619 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/rook/rook-ceph/log to /home/teuthworker/archive/yuriw-2022-08-05_14:49:51-rados-wip-yuri4-testing-2022-08-04-1413-distro-default-smithi/6959556/remote/ubuntu@smithi157.front.sepia.ceph.com/log 2022-08-05T16:25:37.626 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar cz -f - -C /var/lib/rook/rook-ceph/log -- . 2022-08-05T16:25:39.819 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:39.516409 I | op-k8sutil: confirmed csi-cephfsplugin-provisioner does not exist 2022-08-05T16:25:39.820 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:39.523639 I | ceph-csi: successfully removed CSI CephFS driver 2022-08-05T16:25:39.821 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:39.523664 I | ceph-csi: CSI NFS driver disabled 2022-08-05T16:25:39.821 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:39.523676 I | op-k8sutil: removing daemonset csi-nfsplugin if it exists 2022-08-05T16:25:39.821 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:39.529846 I | op-k8sutil: removing deployment csi-nfsplugin-provisioner if it exists 2022-08-05T16:25:39.822 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:39.536132 I | ceph-csi: successfully removed CSI NFS driver 2022-08-05T16:25:41.686 DEBUG:teuthology.misc:Transferring archived files from smithi201:/var/lib/rook/rook-ceph/log to /home/teuthworker/archive/yuriw-2022-08-05_14:49:51-rados-wip-yuri4-testing-2022-08-04-1413-distro-default-smithi/6959556/remote/ubuntu@smithi201.front.sepia.ceph.com/log 2022-08-05T16:25:41.758 DEBUG:teuthology.orchestra.run.smithi201:> sudo tar cz -f - -C /var/lib/rook/rook-ceph/log -- . 2022-08-05T16:25:48.280 ERROR:tasks.rook:'check osd count' reached maximum tries (90) after waiting for 900 seconds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_4a7e6c4ac29de794744a2ce76ca2b36bc637894d/qa/tasks/rook.py", line 163, in rook_operator yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_9e4bf63f00ab3f82a26635c2779f4c3f1b73fb53/teuthology/contextutil.py", line 31, in nested vars.append(enter()) File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_4a7e6c4ac29de794744a2ce76ca2b36bc637894d/qa/tasks/rook.py", line 491, in wait_for_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_9e4bf63f00ab3f82a26635c2779f4c3f1b73fb53/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'check osd count' reached maximum tries (90) after waiting for 900 seconds 2022-08-05T16:25:48.305 INFO:tasks.rook:Cleaning up rook operator 2022-08-05T16:25:48.313 DEBUG:teuthology.orchestra.run.smithi059:> kubectl delete -f operator.yaml 2022-08-05T16:25:48.433 INFO:teuthology.orchestra.run.smithi059.stdout:configmap "rook-ceph-operator-config" deleted 2022-08-05T16:25:48.433 INFO:teuthology.orchestra.run.smithi059.stdout:deployment.apps "rook-ceph-operator" deleted 2022-08-05T16:25:48.463 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:48.415463 I | ceph-csi: CSI Ceph RBD driver disabled 2022-08-05T16:25:48.463 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:48.415500 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists 2022-08-05T16:25:48.464 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:48.435934 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists 2022-08-05T16:25:48.465 DEBUG:teuthology.orchestra.run.smithi059:> rm -rf rook operator.yaml 2022-08-05T16:25:48.543 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:48.462069 I | ceph-csi: successfully removed CSI Ceph RBD driver 2022-08-05T16:25:48.551 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:48.462106 I | ceph-csi: CSI CephFS driver disabled 2022-08-05T16:25:48.570 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:48.462123 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists 2022-08-05T16:25:48.585 INFO:tasks.rook.operator.smithi059.stdout:2022-08-05 16:25:48.497352 I | operator: shutdown signal received, exiting... context canceled 2022-08-05T16:25:49.522 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -rf /var/lib/rook 2022-08-05T16:25:49.649 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf /var/lib/rook 2022-08-05T16:25:49.876 DEBUG:teuthology.orchestra.run.smithi201:> sudo rm -rf /var/lib/rook 2022-08-05T16:25:49.984 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_9e4bf63f00ab3f82a26635c2779f4c3f1b73fb53/teuthology/run_tasks.py", line 106, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_4a7e6c4ac29de794744a2ce76ca2b36bc637894d/qa/tasks/rook.py", line 636, in task lambda: ceph_clients(ctx, config), File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9e4bf63f00ab3f82a26635c2779f4c3f1b73fb53/teuthology/contextutil.py", line 55, in nested raise exc[1] File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_4a7e6c4ac29de794744a2ce76ca2b36bc637894d/qa/tasks/rook.py", line 163, in rook_operator yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_9e4bf63f00ab3f82a26635c2779f4c3f1b73fb53/teuthology/contextutil.py", line 31, in nested vars.append(enter()) File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_4a7e6c4ac29de794744a2ce76ca2b36bc637894d/qa/tasks/rook.py", line 491, in wait_for_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_9e4bf63f00ab3f82a26635c2779f4c3f1b73fb53/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'check osd count' reached maximum tries (90) after waiting for 900 seconds 2022-08-05T16:25:52.289 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=5eab63776bf24f90b03fbebd99c1b567 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_9e4bf63f00ab3f82a26635c2779f4c3f1b73fb53/teuthology/run_tasks.py", line 106, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_4a7e6c4ac29de794744a2ce76ca2b36bc637894d/qa/tasks/rook.py", line 636, in task lambda: ceph_clients(ctx, config), File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9e4bf63f00ab3f82a26635c2779f4c3f1b73fb53/teuthology/contextutil.py", line 55, in nested raise exc[1] File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_4a7e6c4ac29de794744a2ce76ca2b36bc637894d/qa/tasks/rook.py", line 163, in rook_operator yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_9e4bf63f00ab3f82a26635c2779f4c3f1b73fb53/teuthology/contextutil.py", line 31, in nested vars.append(enter()) File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_4a7e6c4ac29de794744a2ce76ca2b36bc637894d/qa/tasks/rook.py", line 491, in wait_for_osds while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_9e4bf63f00ab3f82a26635c2779f4c3f1b73fb53/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'check osd count' reached maximum tries (90) after waiting for 900 seconds 2022-08-05T16:25:52.309 DEBUG:teuthology.run_tasks:Unwinding manager rook 2022-08-05T16:25:52.980 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2022-08-05T16:25:53.594 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi059:/dev/vg_nvme/lv_1... 2022-08-05T16:25:53.636 DEBUG:teuthology.orchestra.run.smithi059:> sudo nvme disconnect -n lv_1 2022-08-05T16:25:54.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-08-05T16:25:54.604 INFO:teuthology.orchestra.run.smithi059.stdout:NQN:lv_1 disconnected 1 controller(s) 2022-08-05T16:25:54.605 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi059:/dev/vg_nvme/lv_2... 2022-08-05T16:25:54.685 DEBUG:teuthology.orchestra.run.smithi059:> sudo nvme disconnect -n lv_2 2022-08-05T16:25:55.029 INFO:teuthology.orchestra.run.smithi059.stdout:NQN:lv_2 disconnected 1 controller(s) 2022-08-05T16:25:55.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-08-05T16:25:55.073 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi059:/dev/vg_nvme/lv_3... 2022-08-05T16:25:55.167 DEBUG:teuthology.orchestra.run.smithi059:> sudo nvme disconnect -n lv_3 2022-08-05T16:25:55.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-08-05T16:25:55.542 INFO:teuthology.orchestra.run.smithi059.stdout:NQN:lv_3 disconnected 1 controller(s) 2022-08-05T16:25:55.606 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi059:/dev/vg_nvme/lv_4... 2022-08-05T16:25:55.607 DEBUG:teuthology.orchestra.run.smithi059:> sudo nvme disconnect -n lv_4 2022-08-05T16:25:55.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-08-05T16:25:55.920 INFO:teuthology.orchestra.run.smithi059.stdout:NQN:lv_4 disconnected 1 controller(s) 2022-08-05T16:25:55.921 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2022-08-05T16:25:55.921 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/scratch_devs 2022-08-05T16:25:56.100 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi157:/dev/vg_nvme/lv_1... 2022-08-05T16:25:56.101 DEBUG:teuthology.orchestra.run.smithi157:> sudo nvme disconnect -n lv_1 2022-08-05T16:25:56.458 INFO:teuthology.orchestra.run.smithi157.stdout:NQN:lv_1 disconnected 1 controller(s) 2022-08-05T16:25:56.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-08-05T16:25:56.461 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi157:/dev/vg_nvme/lv_2... 2022-08-05T16:25:56.498 DEBUG:teuthology.orchestra.run.smithi157:> sudo nvme disconnect -n lv_2 2022-08-05T16:25:56.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-08-05T16:25:56.873 INFO:teuthology.orchestra.run.smithi157.stdout:NQN:lv_2 disconnected 1 controller(s) 2022-08-05T16:25:56.894 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi157:/dev/vg_nvme/lv_3... 2022-08-05T16:25:56.895 DEBUG:teuthology.orchestra.run.smithi157:> sudo nvme disconnect -n lv_3 2022-08-05T16:25:57.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-08-05T16:25:57.262 INFO:teuthology.orchestra.run.smithi157.stdout:NQN:lv_3 disconnected 1 controller(s) 2022-08-05T16:25:57.264 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi157:/dev/vg_nvme/lv_4... 2022-08-05T16:25:57.286 DEBUG:teuthology.orchestra.run.smithi157:> sudo nvme disconnect -n lv_4 2022-08-05T16:25:57.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-08-05T16:25:57.582 INFO:teuthology.orchestra.run.smithi157.stdout:NQN:lv_4 disconnected 1 controller(s) 2022-08-05T16:25:57.583 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2022-08-05T16:25:57.583 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/scratch_devs 2022-08-05T16:25:57.805 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi201:/dev/vg_nvme/lv_1... 2022-08-05T16:25:57.926 DEBUG:teuthology.orchestra.run.smithi201:> sudo nvme disconnect -n lv_1 2022-08-05T16:25:58.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-08-05T16:25:58.297 INFO:teuthology.orchestra.run.smithi201.stdout:NQN:lv_1 disconnected 1 controller(s) 2022-08-05T16:25:58.404 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi201:/dev/vg_nvme/lv_2... 2022-08-05T16:25:58.434 DEBUG:teuthology.orchestra.run.smithi201:> sudo nvme disconnect -n lv_2 2022-08-05T16:25:58.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-08-05T16:25:58.788 INFO:teuthology.orchestra.run.smithi201.stdout:NQN:lv_2 disconnected 1 controller(s) 2022-08-05T16:25:58.789 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi201:/dev/vg_nvme/lv_3... 2022-08-05T16:25:58.793 DEBUG:teuthology.orchestra.run.smithi201:> sudo nvme disconnect -n lv_3 2022-08-05T16:25:59.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-08-05T16:25:59.115 INFO:teuthology.orchestra.run.smithi201.stdout:NQN:lv_3 disconnected 1 controller(s) 2022-08-05T16:25:59.116 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi201:/dev/vg_nvme/lv_4... 2022-08-05T16:25:59.117 DEBUG:teuthology.orchestra.run.smithi201:> sudo nvme disconnect -n lv_4 2022-08-05T16:25:59.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-08-05T16:25:59.461 INFO:teuthology.orchestra.run.smithi201.stdout:NQN:lv_4 disconnected 1 controller(s) 2022-08-05T16:25:59.462 DEBUG:teuthology.orchestra.run.smithi201:> set -ex 2022-08-05T16:25:59.479 DEBUG:teuthology.orchestra.run.smithi201:> sudo dd of=/scratch_devs 2022-08-05T16:25:59.700 DEBUG:teuthology.run_tasks:Unwinding manager kubeadm 2022-08-05T16:26:00.084 INFO:tasks.kubeadm:Tearing down kubeadm 2022-08-05T16:26:00.150 DEBUG:teuthology.orchestra.run.smithi059:> kubectl delete -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml 2022-08-05T16:26:00.474 INFO:teuthology.orchestra.run.smithi059.stdout:namespace "kube-flannel" deleted 2022-08-05T16:26:00.476 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrole.rbac.authorization.k8s.io "flannel" deleted 2022-08-05T16:26:00.476 INFO:teuthology.orchestra.run.smithi059.stdout:clusterrolebinding.rbac.authorization.k8s.io "flannel" deleted 2022-08-05T16:26:00.477 INFO:teuthology.orchestra.run.smithi059.stdout:serviceaccount "flannel" deleted 2022-08-05T16:26:00.479 INFO:teuthology.orchestra.run.smithi059.stdout:configmap "kube-flannel-cfg" deleted 2022-08-05T16:26:00.479 INFO:teuthology.orchestra.run.smithi059.stdout:daemonset.apps "kube-flannel-ds" deleted 2022-08-05T16:26:06.213 INFO:tasks.kubeadm:Deconfiguring kubectl 2022-08-05T16:26:06.214 DEBUG:teuthology.orchestra.run.smithi059:> rm -rf .kube && sudo rm -rf /root/.kube 2022-08-05T16:26:06.251 DEBUG:teuthology.orchestra.run.smithi157:> rm -rf .kube && sudo rm -rf /root/.kube 2022-08-05T16:26:06.332 DEBUG:teuthology.orchestra.run.smithi201:> rm -rf .kube && sudo rm -rf /root/.kube 2022-08-05T16:26:06.423 INFO:tasks.kubeadm:Cleaning up node 2022-08-05T16:26:06.533 DEBUG:teuthology.orchestra.run.smithi059:> sudo kubeadm reset cleanup-node -f 2022-08-05T16:26:06.681 DEBUG:teuthology.orchestra.run.smithi157:> sudo kubeadm reset cleanup-node -f 2022-08-05T16:26:06.884 INFO:teuthology.orchestra.run.smithi059.stdout:[reset] Reading configuration from the cluster... 2022-08-05T16:26:06.964 INFO:teuthology.orchestra.run.smithi059.stdout:[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' 2022-08-05T16:26:06.965 INFO:teuthology.orchestra.run.smithi059.stdout:[preflight] Running pre-flight checks 2022-08-05T16:26:06.965 INFO:teuthology.orchestra.run.smithi059.stdout:[reset] Stopping the kubelet service 2022-08-05T16:26:06.965 INFO:teuthology.orchestra.run.smithi059.stdout:[reset] Unmounting mounted directories in "/var/lib/kubelet" 2022-08-05T16:26:06.988 DEBUG:teuthology.orchestra.run.smithi201:> sudo kubeadm reset cleanup-node -f 2022-08-05T16:26:06.989 INFO:teuthology.orchestra.run.smithi157.stdout:[preflight] Running pre-flight checks 2022-08-05T16:26:06.990 INFO:teuthology.orchestra.run.smithi157.stdout:[reset] No etcd config found. Assuming external etcd 2022-08-05T16:26:06.990 INFO:teuthology.orchestra.run.smithi157.stdout:[reset] Please, manually reset etcd to prevent further issues 2022-08-05T16:26:07.077 INFO:teuthology.orchestra.run.smithi157.stdout:[reset] Stopping the kubelet service 2022-08-05T16:26:07.125 INFO:teuthology.orchestra.run.smithi157.stderr:W0805 16:26:06.872340 46225 removeetcdmember.go:84] [reset] No kubeadm config, using etcd pod spec to get data directory 2022-08-05T16:26:07.126 INFO:teuthology.orchestra.run.smithi157.stdout:[reset] Unmounting mounted directories in "/var/lib/kubelet" 2022-08-05T16:26:07.188 INFO:teuthology.orchestra.run.smithi201.stdout:[preflight] Running pre-flight checks 2022-08-05T16:26:07.210 INFO:teuthology.orchestra.run.smithi201.stderr:W0805 16:26:07.174241 49249 removeetcdmember.go:84] [reset] No kubeadm config, using etcd pod spec to get data directory 2022-08-05T16:26:07.211 INFO:teuthology.orchestra.run.smithi201.stdout:[reset] No etcd config found. Assuming external etcd 2022-08-05T16:26:07.212 INFO:teuthology.orchestra.run.smithi201.stdout:[reset] Please, manually reset etcd to prevent further issues 2022-08-05T16:26:07.212 INFO:teuthology.orchestra.run.smithi201.stdout:[reset] Stopping the kubelet service 2022-08-05T16:26:07.261 INFO:teuthology.orchestra.run.smithi201.stdout:[reset] Unmounting mounted directories in "/var/lib/kubelet" 2022-08-05T16:26:08.598 INFO:teuthology.orchestra.run.smithi201.stdout:[reset] Deleting contents of directories: [/etc/kubernetes/manifests /etc/kubernetes/pki] 2022-08-05T16:26:08.599 INFO:teuthology.orchestra.run.smithi201.stdout:[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf] 2022-08-05T16:26:08.661 INFO:teuthology.orchestra.run.smithi201.stdout:[reset] Deleting contents of stateful directories: [/var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni] 2022-08-05T16:26:08.692 INFO:teuthology.orchestra.run.smithi201.stdout: 2022-08-05T16:26:08.693 INFO:teuthology.orchestra.run.smithi201.stdout:The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d 2022-08-05T16:26:08.693 INFO:teuthology.orchestra.run.smithi201.stdout: 2022-08-05T16:26:08.693 INFO:teuthology.orchestra.run.smithi201.stdout:The reset process does not reset or clean up iptables rules or IPVS tables. 2022-08-05T16:26:08.747 INFO:teuthology.orchestra.run.smithi201.stdout:If you wish to reset iptables, you must do so manually by using the "iptables" command. 2022-08-05T16:26:08.748 INFO:teuthology.orchestra.run.smithi201.stdout: 2022-08-05T16:26:08.748 INFO:teuthology.orchestra.run.smithi201.stdout:If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar) 2022-08-05T16:26:08.749 INFO:teuthology.orchestra.run.smithi201.stdout:to reset your system's IPVS tables. 2022-08-05T16:26:08.749 INFO:teuthology.orchestra.run.smithi201.stdout: 2022-08-05T16:26:08.749 INFO:teuthology.orchestra.run.smithi201.stdout:The reset process does not clean your kubeconfig files and you must remove them manually. 2022-08-05T16:26:08.769 INFO:teuthology.orchestra.run.smithi201.stdout:Please, check the contents of the $HOME/.kube/config file. 2022-08-05T16:26:10.336 INFO:teuthology.orchestra.run.smithi157.stdout:[reset] Deleting contents of directories: [/etc/kubernetes/manifests /etc/kubernetes/pki] 2022-08-05T16:26:10.336 INFO:teuthology.orchestra.run.smithi157.stdout:[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf] 2022-08-05T16:26:10.337 INFO:teuthology.orchestra.run.smithi157.stdout:[reset] Deleting contents of stateful directories: [/var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni] 2022-08-05T16:26:10.337 INFO:teuthology.orchestra.run.smithi157.stdout: 2022-08-05T16:26:10.338 INFO:teuthology.orchestra.run.smithi157.stdout:The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d 2022-08-05T16:26:10.356 INFO:teuthology.orchestra.run.smithi157.stdout: 2022-08-05T16:26:10.357 INFO:teuthology.orchestra.run.smithi157.stdout:The reset process does not reset or clean up iptables rules or IPVS tables. 2022-08-05T16:26:10.357 INFO:teuthology.orchestra.run.smithi157.stdout:If you wish to reset iptables, you must do so manually by using the "iptables" command. 2022-08-05T16:26:10.357 INFO:teuthology.orchestra.run.smithi157.stdout: 2022-08-05T16:26:10.374 INFO:teuthology.orchestra.run.smithi157.stdout:If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar) 2022-08-05T16:26:10.374 INFO:teuthology.orchestra.run.smithi157.stdout:to reset your system's IPVS tables. 2022-08-05T16:26:10.375 INFO:teuthology.orchestra.run.smithi157.stdout: 2022-08-05T16:26:10.385 INFO:teuthology.orchestra.run.smithi157.stdout:The reset process does not clean your kubeconfig files and you must remove them manually. 2022-08-05T16:26:10.385 INFO:teuthology.orchestra.run.smithi157.stdout:Please, check the contents of the $HOME/.kube/config file. 2022-08-05T16:26:10.940 INFO:teuthology.orchestra.run.smithi059.stdout:[reset] Deleting contents of directories: [/etc/kubernetes/manifests /etc/kubernetes/pki] 2022-08-05T16:26:10.941 INFO:teuthology.orchestra.run.smithi059.stdout:[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf] 2022-08-05T16:26:10.964 INFO:teuthology.orchestra.run.smithi059.stdout:[reset] Deleting contents of stateful directories: [/var/lib/etcd /var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni] 2022-08-05T16:26:10.965 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:26:10.990 INFO:teuthology.orchestra.run.smithi059.stdout:The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d 2022-08-05T16:26:11.007 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:26:11.008 INFO:teuthology.orchestra.run.smithi059.stdout:The reset process does not reset or clean up iptables rules or IPVS tables. 2022-08-05T16:26:11.009 INFO:teuthology.orchestra.run.smithi059.stdout:If you wish to reset iptables, you must do so manually by using the "iptables" command. 2022-08-05T16:26:11.020 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:26:11.020 INFO:teuthology.orchestra.run.smithi059.stdout:If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar) 2022-08-05T16:26:11.021 INFO:teuthology.orchestra.run.smithi059.stdout:to reset your system's IPVS tables. 2022-08-05T16:26:11.121 INFO:teuthology.orchestra.run.smithi059.stdout: 2022-08-05T16:26:11.144 INFO:teuthology.orchestra.run.smithi059.stdout:The reset process does not clean your kubeconfig files and you must remove them manually. 2022-08-05T16:26:11.146 INFO:teuthology.orchestra.run.smithi059.stdout:Please, check the contents of the $HOME/.kube/config file. 2022-08-05T16:26:11.147 INFO:tasks.kubeadm:Uninstalling kube{adm,let,ctl} 2022-08-05T16:26:11.149 DEBUG:teuthology.run_tasks:Unwinding manager clock 2022-08-05T16:26:11.385 INFO:teuthology.task.clock:Checking final clock skew... 2022-08-05T16:26:11.451 DEBUG:teuthology.orchestra.run.smithi059:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-08-05T16:26:11.608 DEBUG:teuthology.orchestra.run.smithi157:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-08-05T16:26:11.745 INFO:teuthology.orchestra.run.smithi059.stdout: remote refid st t when poll reach delay offset jitter 2022-08-05T16:26:11.834 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================== 2022-08-05T16:26:11.835 INFO:teuthology.orchestra.run.smithi059.stdout:+hv01.front.sepi 216.240.36.24 3 u 27 64 377 0.095 0.071 0.276 2022-08-05T16:26:11.835 INFO:teuthology.orchestra.run.smithi059.stdout:*hv02.front.sepi 163.237.218.18 2 u 43 64 377 0.101 2.442 0.100 2022-08-05T16:26:11.864 INFO:teuthology.orchestra.run.smithi059.stdout:+hv03.front.sepi 216.229.4.69 3 u 32 64 377 0.078 0.301 0.134 2022-08-05T16:26:11.872 INFO:teuthology.orchestra.run.smithi059.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 0.000 0.000 2022-08-05T16:26:12.064 DEBUG:teuthology.orchestra.run.smithi201:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-08-05T16:26:12.181 INFO:teuthology.orchestra.run.smithi157.stdout: remote refid st t when poll reach delay offset jitter 2022-08-05T16:26:12.181 INFO:teuthology.orchestra.run.smithi157.stdout:============================================================================== 2022-08-05T16:26:12.307 INFO:teuthology.orchestra.run.smithi157.stdout:+hv01.front.sepi 216.240.36.24 3 u 22 64 377 0.085 -2.359 0.580 2022-08-05T16:26:12.308 INFO:teuthology.orchestra.run.smithi157.stdout:*hv02.front.sepi 163.237.218.18 2 u 25 64 377 0.079 0.042 0.374 2022-08-05T16:26:12.308 INFO:teuthology.orchestra.run.smithi157.stdout:+hv03.front.sepi 216.229.4.69 3 u 38 64 377 0.097 -2.711 0.674 2022-08-05T16:26:12.309 INFO:teuthology.orchestra.run.smithi157.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 0.000 0.000 2022-08-05T16:26:12.746 INFO:teuthology.orchestra.run.smithi201.stdout: remote refid st t when poll reach delay offset jitter 2022-08-05T16:26:12.817 INFO:teuthology.orchestra.run.smithi201.stdout:============================================================================== 2022-08-05T16:26:12.860 INFO:teuthology.orchestra.run.smithi201.stdout:+hv01.front.sepi 216.240.36.24 3 u 24 64 377 0.076 -1.893 0.798 2022-08-05T16:26:12.877 INFO:teuthology.orchestra.run.smithi201.stdout:*hv02.front.sepi 163.237.218.18 2 u 25 64 377 0.089 0.663 0.453 2022-08-05T16:26:12.897 INFO:teuthology.orchestra.run.smithi201.stdout:+hv03.front.sepi 216.229.4.69 3 u 26 64 377 0.096 -0.937 0.722 2022-08-05T16:26:12.898 INFO:teuthology.orchestra.run.smithi201.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 0.000 0.000 2022-08-05T16:26:12.932 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2022-08-05T16:26:13.579 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2022-08-05T16:26:13.710 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2022-08-05T16:26:14.178 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2022-08-05T16:26:14.431 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2022-08-05T16:26:15.235 INFO:teuthology.task.internal:Duration was 1997.132642 seconds 2022-08-05T16:26:15.462 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2022-08-05T16:26:16.194 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2022-08-05T16:26:16.202 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-08-05T16:26:16.328 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-08-05T16:26:16.548 DEBUG:teuthology.orchestra.run.smithi201:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-08-05T16:26:16.639 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2022-08-05T16:26:16.639 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi059.front.sepia.ceph.com 2022-08-05T16:26:16.725 DEBUG:teuthology.orchestra.run.smithi059:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-08-05T16:26:16.818 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi157.front.sepia.ceph.com 2022-08-05T16:26:16.819 DEBUG:teuthology.orchestra.run.smithi157:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-08-05T16:26:17.031 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi201.front.sepia.ceph.com 2022-08-05T16:26:17.032 DEBUG:teuthology.orchestra.run.smithi201:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-08-05T16:26:17.179 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2022-08-05T16:26:17.180 DEBUG:teuthology.orchestra.run.smithi059:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-08-05T16:26:17.375 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-08-05T16:26:17.511 DEBUG:teuthology.orchestra.run.smithi201:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-08-05T16:26:17.888 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2022-08-05T16:26:17.889 DEBUG:teuthology.orchestra.run.smithi059:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-08-05T16:26:17.942 DEBUG:teuthology.orchestra.run.smithi157:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-08-05T16:26:18.082 DEBUG:teuthology.orchestra.run.smithi201:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-08-05T16:26:18.265 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2022-08-05T16:26:18.757 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2022-08-05T16:26:18.978 DEBUG:teuthology.orchestra.run.smithi059:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-08-05T16:26:19.109 DEBUG:teuthology.orchestra.run.smithi157:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-08-05T16:26:19.516 DEBUG:teuthology.orchestra.run.smithi201:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-08-05T16:26:19.643 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2022-08-05T16:26:20.045 DEBUG:teuthology.orchestra.run.smithi059:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-08-05T16:26:20.059 DEBUG:teuthology.orchestra.run.smithi157:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-08-05T16:26:20.130 DEBUG:teuthology.orchestra.run.smithi201:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-08-05T16:26:20.168 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = core 2022-08-05T16:26:20.277 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = core 2022-08-05T16:26:20.321 INFO:teuthology.orchestra.run.smithi201.stdout:kernel.core_pattern = core 2022-08-05T16:26:20.420 DEBUG:teuthology.orchestra.run.smithi059:> test -e /home/ubuntu/cephtest/archive/coredump 2022-08-05T16:26:20.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-08-05T16:26:20.677 DEBUG:teuthology.orchestra.run.smithi157:> test -e /home/ubuntu/cephtest/archive/coredump 2022-08-05T16:26:20.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-08-05T16:26:20.955 DEBUG:teuthology.orchestra.run.smithi201:> test -e /home/ubuntu/cephtest/archive/coredump 2022-08-05T16:26:21.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-08-05T16:26:21.091 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2022-08-05T16:26:21.292 INFO:teuthology.task.internal:Transferring archived files... 2022-08-05T16:26:21.402 DEBUG:teuthology.misc:Transferring archived files from smithi059:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-08-05_14:49:51-rados-wip-yuri4-testing-2022-08-04-1413-distro-default-smithi/6959556/remote/smithi059 2022-08-05T16:26:21.495 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-08-05T16:26:21.837 DEBUG:teuthology.misc:Transferring archived files from smithi157:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-08-05_14:49:51-rados-wip-yuri4-testing-2022-08-04-1413-distro-default-smithi/6959556/remote/smithi157 2022-08-05T16:26:21.843 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-08-05T16:26:22.328 DEBUG:teuthology.misc:Transferring archived files from smithi201:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-08-05_14:49:51-rados-wip-yuri4-testing-2022-08-04-1413-distro-default-smithi/6959556/remote/smithi201 2022-08-05T16:26:22.354 DEBUG:teuthology.orchestra.run.smithi201:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-08-05T16:26:22.681 INFO:teuthology.task.internal:Removing archive directory... 2022-08-05T16:26:22.695 DEBUG:teuthology.orchestra.run.smithi059:> rm -rf -- /home/ubuntu/cephtest/archive 2022-08-05T16:26:22.758 DEBUG:teuthology.orchestra.run.smithi157:> rm -rf -- /home/ubuntu/cephtest/archive 2022-08-05T16:26:22.946 DEBUG:teuthology.orchestra.run.smithi201:> rm -rf -- /home/ubuntu/cephtest/archive 2022-08-05T16:26:23.189 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2022-08-05T16:26:23.375 INFO:teuthology.task.internal:Not uploading archives. 2022-08-05T16:26:23.377 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2022-08-05T16:26:23.585 INFO:teuthology.task.internal:Tidying up after the test... 2022-08-05T16:26:23.588 DEBUG:teuthology.orchestra.run.smithi059:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-08-05T16:26:23.739 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-08-05T16:26:23.820 INFO:teuthology.orchestra.run.smithi059.stdout: 524315 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 5 16:26 /home/ubuntu/cephtest 2022-08-05T16:26:23.827 DEBUG:teuthology.orchestra.run.smithi201:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-08-05T16:26:24.031 INFO:teuthology.orchestra.run.smithi157.stdout: 524315 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 5 16:26 /home/ubuntu/cephtest 2022-08-05T16:26:24.095 INFO:teuthology.orchestra.run.smithi201.stdout: 524315 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 5 16:26 /home/ubuntu/cephtest 2022-08-05T16:26:24.126 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2022-08-05T16:26:24.734 INFO:teuthology.nuke:Checking targets against current locks 2022-08-05T16:26:25.216 DEBUG:teuthology.nuke:shortname: smithi059 2022-08-05T16:26:25.236 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-08-05T16:26:25.611 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-08-05_14:49:51-rados-wip-yuri4-testing-2022-08-04-1413-distro-default-smithi/6959556', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-08-05 15:43:03.313479', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwea/Iirc1kzi7g4++3ZYg8ZzcV3HAIL0a2QPGSzHOXLlNjv5Aw83/K2S++7Oc5xnaKjfiT88I56xzTKbZ3NOTPMQrz7hFnSgzObbILWZJnlngTCePPrD0dLn/MEjXVTx/WaVCawO5x1ygswCcEjV3OUOwx8Z9DqL2LbcfYd4ZseDczLnpQoBIktBNiCtG7kvSLVLNXRtzAa2xIos3boSwTciOppVt4O9CKPseDAPF3U+c2fOWz/Vf2tV39ahhaDGPSv2wQv08YbLFq7M8fSdxNSFKzf8Suaipyz3DFYsjkDmxapmHnOltbr0dyM7X37sVPF/nBgvS+tjwUZJo5qyrk2tg/GFlkZOwPURUqlxKFq0udDQdnyPQvjpKh2mExpLi6rN+do2OMxgSFxPmbxP6RrRlUjEyks56u/oQ2rolA/znTrG5wXE0kq6znzboz3a/9pBtBRQGiWpcoz+mGEySC6o2+A+LrAT0I5OX0kDepZoKvoaGE3L0Ez0NayJ8n3M='} 2022-08-05T16:26:25.683 DEBUG:teuthology.nuke:shortname: smithi157 2022-08-05T16:26:25.684 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-08-05T16:26:25.997 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-08-05_14:49:51-rados-wip-yuri4-testing-2022-08-04-1413-distro-default-smithi/6959556', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-08-05 15:43:03.314986', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCnVoKVrn3sZuiTXZEQz6r73J9dDTIqiTFapvnv8+CciqE+0IBzmcqTH62uzOJPJ9UFBigiWm0paY33kJ7QSu5ddaqPT5joAtxhFg8IIlM1lvqYjbDbgaxBC6/8tsxRppsWYWpbP3eyBEQWanpat7OLk6cr+zIivzEHneQbmBdsEtjb/ii8SlB/SSTRoI/e52/TUM9uLunIReCoyWDwZpVxerz+v1DHO96VMYgtxx71v9vs2HeNJaJ62Bz5dRo3eWpfQ9Rbd0A0KcGZmbhov6TLQc8ycF2rn8388ahgpxl0ZgUllQdJiZ+bLHfLtGNgV3xOSaceu4LPDG6qkMKafXb82AT3z3O1PraHxfg4syHFBbfEXpHm8XwD3OlSJzJ0X5zc6sYBaCfmu3QOhtnNT+K9QdPVkV2iuQN+pzUXv/pMmUMPmCjLH76FgtaxAwNOczusoRF9qGR3jcxCVbsCc//2/f7ZoPrqnWW3r7CilBibEtvHSl1vaA2UxnB6hOBvCL8='} 2022-08-05T16:26:26.101 DEBUG:teuthology.nuke:shortname: smithi201 2022-08-05T16:26:26.117 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-08-05T16:26:26.321 INFO:teuthology.orchestra.console:Power off smithi059 2022-08-05T16:26:26.323 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2022-08-05T16:26:26.533 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi201.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-08-05_14:49:51-rados-wip-yuri4-testing-2022-08-04-1413-distro-default-smithi/6959556', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-08-05 15:43:03.316038', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6BWqMWMl/8HUTPMu35Ryd9ydQYtDXEofXrP4adLcwMgGoIbCjT0I4/XVJ8Z6EGysUqGnzFq0EdDuhWIsLd5w16gKVg+zo0LqjFGWMIv2fEu9CRCqZZfesyGNJZ0jJlpRcwXFG4J/5eppES3gfrCZ5Dz4hgHzavS7ST3zk1tSATPp111t/nzWdPaIWGjVVujUIRQEM4hCqe/qow9DXUwmRhDNqcoAU6Wz0DYplsR4eyoDBBWSN++Ro6BCNFmOGR2HhdT0d0sCFAzMJNjLRQQoEOOrQRBh9TwKlGdkqdazWVH591mOLBIY8CzyyWrVFZwrWYNkjt3s7H+TsQasHjfuv8shbJPwLvimqsiKphMc14ca5durzw0lTERRXnaR75huj3CzoYGsdnzeXya4lWzAlxtXV0ZMTz7nnAOSgf+Pl6k4S+jJEpKIKGvTskakR7dWDUEz+b0hKzXEv0ihSE46dJXjwoFjTljzWXWZzvkoeP1apJ0mjTvC5WPf2eo10sYU='} 2022-08-05T16:26:26.819 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-08-05T16:26:28.009 INFO:teuthology.orchestra.console:Power off smithi157 2022-08-05T16:26:28.011 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2022-08-05T16:26:28.310 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-08-05T16:26:29.256 INFO:teuthology.orchestra.console:Power off smithi201 2022-08-05T16:26:29.258 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi201.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2022-08-05T16:26:29.393 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi201.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-08-05T16:26:32.072 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-08-05T16:26:33.379 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-08-05T16:26:33.718 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi201.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-08-05T16:26:36.352 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-08-05T16:26:36.662 INFO:teuthology.orchestra.console:Power off for smithi059 completed 2022-08-05T16:26:37.634 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-08-05T16:26:37.968 INFO:teuthology.orchestra.console:Power off for smithi157 completed 2022-08-05T16:26:38.126 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi201.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2022-08-05T16:26:38.479 INFO:teuthology.orchestra.console:Power off for smithi201 completed 2022-08-05T16:26:38.673 INFO:teuthology.run:Summary data: description: rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/3-node k8s/1.21 net/flannel rook/master} duration: 1997.1326422691345 failure_reason: '''check osd count'' reached maximum tries (90) after waiting for 900 seconds' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=5eab63776bf24f90b03fbebd99c1b567 status: fail success: false 2022-08-05T16:26:38.676 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-08-05T16:26:39.265 INFO:teuthology.run:FAIL