2022-01-26T04:14:38.062 INFO:root:teuthology version: 1.1.0-7c0cb867 2022-01-26T04:14:38.221 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi002.front.sepia.ceph.com' reimaging 2022-01-26T04:14:38.222 INFO:teuthology.lock.ops:Updating [ubuntu@smithi002.front.sepia.ceph.com]: reset os type and version on server 2022-01-26T04:14:38.222 INFO:teuthology.lock.ops:Updating smithi002.front.sepia.ceph.com on lock server 2022-01-26T04:14:38.240 INFO:teuthology.lock.ops:Node 'ubuntu@smithi002.front.sepia.ceph.com' reimaging is complete 2022-01-26T04:14:38.240 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi023.front.sepia.ceph.com' reimaging 2022-01-26T04:14:38.240 INFO:teuthology.lock.ops:Updating [ubuntu@smithi023.front.sepia.ceph.com]: reset os type and version on server 2022-01-26T04:14:38.241 INFO:teuthology.lock.ops:Updating smithi023.front.sepia.ceph.com on lock server 2022-01-26T04:14:38.260 INFO:teuthology.lock.ops:Node 'ubuntu@smithi023.front.sepia.ceph.com' reimaging is complete 2022-01-26T04:14:38.260 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi084.front.sepia.ceph.com' reimaging 2022-01-26T04:14:38.260 INFO:teuthology.lock.ops:Updating [ubuntu@smithi084.front.sepia.ceph.com]: reset os type and version on server 2022-01-26T04:14:38.261 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2022-01-26T04:14:38.279 INFO:teuthology.lock.ops:Node 'ubuntu@smithi084.front.sepia.ceph.com' reimaging is complete 2022-01-26T04:14:38.487 INFO:teuthology.provision.fog.smithi002:Scheduling deploy of ubuntu 20.04 2022-01-26T04:14:38.491 INFO:teuthology.provision.fog.smithi023:Scheduling deploy of ubuntu 20.04 2022-01-26T04:14:38.511 INFO:teuthology.provision.fog.smithi084:Scheduling deploy of ubuntu 20.04 2022-01-26T04:14:38.893 INFO:teuthology.orchestra.console:Power off smithi023 2022-01-26T04:14:38.929 INFO:teuthology.orchestra.console:Power off smithi002 2022-01-26T04:14:38.943 INFO:teuthology.orchestra.console:Power off smithi084 2022-01-26T04:14:47.294 INFO:teuthology.orchestra.console:Power off for smithi002 completed 2022-01-26T04:14:47.305 INFO:teuthology.orchestra.console:Power off for smithi084 completed 2022-01-26T04:14:47.396 INFO:teuthology.orchestra.console:Power on smithi002 2022-01-26T04:14:47.406 INFO:teuthology.orchestra.console:Power on smithi084 2022-01-26T04:14:51.384 INFO:teuthology.orchestra.console:Power off for smithi023 completed 2022-01-26T04:14:51.486 INFO:teuthology.orchestra.console:Power on smithi023 2022-01-26T04:14:55.741 INFO:teuthology.orchestra.console:Power on for smithi023 completed 2022-01-26T04:14:55.752 INFO:teuthology.orchestra.console:Power on for smithi002 completed 2022-01-26T04:14:55.766 INFO:teuthology.orchestra.console:Power on for smithi084 completed 2022-01-26T04:14:55.842 INFO:teuthology.provision.fog.smithi023:Waiting for deploy to finish 2022-01-26T04:14:55.853 INFO:teuthology.provision.fog.smithi002:Waiting for deploy to finish 2022-01-26T04:14:55.867 INFO:teuthology.provision.fog.smithi084:Waiting for deploy to finish 2022-01-26T04:19:33.636 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-01-26T04:19:34.170 INFO:teuthology.provision.fog.smithi002:Node is ready 2022-01-26T04:19:34.177 INFO:teuthology.orchestra.run.smithi002.stdout:smithi002.front.sepia.ceph.com 2022-01-26T04:19:34.184 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2 smithi002.front.sepia.ceph.com smithi002 2022-01-26T04:19:34.419 INFO:teuthology.provision.fog.smithi002:Deploy complete! 2022-01-26T04:19:59.455 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-01-26T04:19:59.955 INFO:teuthology.provision.fog.smithi084:Node is ready 2022-01-26T04:20:00.003 INFO:teuthology.orchestra.run.smithi084.stdout:smithi084.front.sepia.ceph.com 2022-01-26T04:20:00.011 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84 smithi084.front.sepia.ceph.com smithi084 2022-01-26T04:20:00.258 INFO:teuthology.provision.fog.smithi084:Deploy complete! 2022-01-26T04:20:46.032 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-01-26T04:20:46.534 INFO:teuthology.provision.fog.smithi023:Node is ready 2022-01-26T04:20:46.540 INFO:teuthology.orchestra.run.smithi023.stdout:smithi023.front.sepia.ceph.com 2022-01-26T04:20:46.548 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23 smithi023.front.sepia.ceph.com smithi023 2022-01-26T04:20:46.752 INFO:teuthology.provision.fog.smithi023:Deploy complete! 2022-01-26T04:20:46.981 INFO:teuthology.lock.ops:Checking smithi002.front.sepia.ceph.com 2022-01-26T04:20:46.982 INFO:teuthology.lock.ops:New key found. Updating... 2022-01-26T04:20:47.003 INFO:teuthology.lock.ops:Checking smithi023.front.sepia.ceph.com 2022-01-26T04:20:47.003 INFO:teuthology.lock.ops:New key found. Updating... 2022-01-26T04:20:47.021 INFO:teuthology.lock.ops:Checking smithi084.front.sepia.ceph.com 2022-01-26T04:20:47.022 INFO:teuthology.lock.ops:New key found. Updating... 2022-01-26T04:20:47.042 INFO:teuthology.lock.ops:Updating [smithi002.front.sepia.ceph.com]: set os type and version on server 2022-01-26T04:20:47.043 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2022-01-26T04:20:47.318 INFO:teuthology.orchestra.run.smithi002.stdout:x86_64 2022-01-26T04:20:47.325 INFO:teuthology.orchestra.run.smithi002.stdout:NAME="Ubuntu" 2022-01-26T04:20:47.325 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-01-26T04:20:47.325 INFO:teuthology.orchestra.run.smithi002.stdout:ID=ubuntu 2022-01-26T04:20:47.325 INFO:teuthology.orchestra.run.smithi002.stdout:ID_LIKE=debian 2022-01-26T04:20:47.326 INFO:teuthology.orchestra.run.smithi002.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-01-26T04:20:47.326 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION_ID="20.04" 2022-01-26T04:20:47.326 INFO:teuthology.orchestra.run.smithi002.stdout:HOME_URL="https://www.ubuntu.com/" 2022-01-26T04:20:47.326 INFO:teuthology.orchestra.run.smithi002.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-01-26T04:20:47.327 INFO:teuthology.orchestra.run.smithi002.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-01-26T04:20:47.327 INFO:teuthology.orchestra.run.smithi002.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-01-26T04:20:47.327 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION_CODENAME=focal 2022-01-26T04:20:47.327 INFO:teuthology.orchestra.run.smithi002.stdout:UBUNTU_CODENAME=focal 2022-01-26T04:20:47.329 INFO:teuthology.lock.ops:Updating smithi002.front.sepia.ceph.com on lock server 2022-01-26T04:20:47.348 INFO:teuthology.lock.ops:Updating [smithi023.front.sepia.ceph.com]: set os type and version on server 2022-01-26T04:20:47.349 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2022-01-26T04:20:47.514 INFO:teuthology.orchestra.run.smithi023.stdout:x86_64 2022-01-26T04:20:47.562 INFO:teuthology.orchestra.run.smithi023.stdout:NAME="Ubuntu" 2022-01-26T04:20:47.562 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-01-26T04:20:47.562 INFO:teuthology.orchestra.run.smithi023.stdout:ID=ubuntu 2022-01-26T04:20:47.563 INFO:teuthology.orchestra.run.smithi023.stdout:ID_LIKE=debian 2022-01-26T04:20:47.563 INFO:teuthology.orchestra.run.smithi023.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-01-26T04:20:47.563 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION_ID="20.04" 2022-01-26T04:20:47.563 INFO:teuthology.orchestra.run.smithi023.stdout:HOME_URL="https://www.ubuntu.com/" 2022-01-26T04:20:47.564 INFO:teuthology.orchestra.run.smithi023.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-01-26T04:20:47.564 INFO:teuthology.orchestra.run.smithi023.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-01-26T04:20:47.564 INFO:teuthology.orchestra.run.smithi023.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-01-26T04:20:47.564 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION_CODENAME=focal 2022-01-26T04:20:47.564 INFO:teuthology.orchestra.run.smithi023.stdout:UBUNTU_CODENAME=focal 2022-01-26T04:20:47.565 INFO:teuthology.lock.ops:Updating smithi023.front.sepia.ceph.com on lock server 2022-01-26T04:20:47.583 INFO:teuthology.lock.ops:Updating [smithi084.front.sepia.ceph.com]: set os type and version on server 2022-01-26T04:20:47.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2022-01-26T05:21:47.875 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/dispatcher/supervisor.py", line 206, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/lock/ops.py", line 317, in reimage_machines update_nodes(reimaged) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/lock/ops.py", line 38, in update_nodes inventory_info = remote.inventory_info File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/remote.py", line 663, in inventory_info node['arch'] = self.arch File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/remote.py", line 320, in arch self._arch = self.sh('uname -m').strip() File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 508, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 879, in open_session timeout=timeout, File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 1010, in open_channel raise SSHException("Timeout opening channel.") paramiko.ssh_exception.SSHException: Timeout opening channel. 2022-01-26T05:21:47.986 INFO:teuthology.nuke:Checking targets against current locks 2022-01-26T05:21:48.133 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-01-26T05:21:48.140 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-01-26T05:21:48.144 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-01-26T05:21:48.215 INFO:teuthology.orchestra.console:Power off smithi084 2022-01-26T05:21:48.260 INFO:teuthology.orchestra.console:Power off smithi002 2022-01-26T05:21:48.279 INFO:teuthology.orchestra.console:Power off smithi023 2022-01-26T05:21:56.643 INFO:teuthology.orchestra.console:Power off for smithi002 completed 2022-01-26T05:21:56.652 INFO:teuthology.orchestra.console:Power off for smithi023 completed 2022-01-26T05:21:56.806 INFO:teuthology.lock.ops:unlocked: smithi023.front.sepia.ceph.com 2022-01-26T05:21:56.811 INFO:teuthology.lock.ops:unlocked: smithi002.front.sepia.ceph.com 2022-01-26T05:22:00.735 INFO:teuthology.orchestra.console:Power off for smithi084 completed 2022-01-26T05:22:00.884 INFO:teuthology.lock.ops:unlocked: smithi084.front.sepia.ceph.com 2022-01-26T05:22:01.175 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/bin/teuthology-dispatcher", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-dispatcher')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/scripts/dispatcher.py", line 33, in main teuthology.dispatcher.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/dispatcher/__init__.py", line 61, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/dispatcher/supervisor.py", line 52, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/dispatcher/supervisor.py", line 206, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/lock/ops.py", line 317, in reimage_machines update_nodes(reimaged) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/lock/ops.py", line 38, in update_nodes inventory_info = remote.inventory_info File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/remote.py", line 663, in inventory_info node['arch'] = self.arch File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/remote.py", line 320, in arch self._arch = self.sh('uname -m').strip() File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 508, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 879, in open_session timeout=timeout, File "/home/teuthworker/src/git.ceph.com_git_teuthology_7c0cb8672986d9dbe53078a123af65593653ef7a/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 1010, in open_channel raise SSHException("Timeout opening channel.") paramiko.ssh_exception.SSHException: Timeout opening channel.