2021-06-22T23:15:11.756 INFO:root:teuthology version: 1.1.0-f359b10d 2021-06-22T23:15:11.864 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba010.front.sepia.ceph.com' reimaging 2021-06-22T23:15:11.865 INFO:teuthology.lock.ops:Updating [ubuntu@gibba010.front.sepia.ceph.com]: reset os type and version on server 2021-06-22T23:15:11.866 INFO:teuthology.lock.ops:Updating gibba010.front.sepia.ceph.com on lock server 2021-06-22T23:15:11.880 INFO:teuthology.lock.ops:Node 'ubuntu@gibba010.front.sepia.ceph.com' reimaging is complete 2021-06-22T23:15:11.880 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba028.front.sepia.ceph.com' reimaging 2021-06-22T23:15:11.880 INFO:teuthology.lock.ops:Updating [ubuntu@gibba028.front.sepia.ceph.com]: reset os type and version on server 2021-06-22T23:15:11.881 INFO:teuthology.lock.ops:Updating gibba028.front.sepia.ceph.com on lock server 2021-06-22T23:15:11.895 INFO:teuthology.lock.ops:Node 'ubuntu@gibba028.front.sepia.ceph.com' reimaging is complete 2021-06-22T23:15:11.895 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba030.front.sepia.ceph.com' reimaging 2021-06-22T23:15:11.895 INFO:teuthology.lock.ops:Updating [ubuntu@gibba030.front.sepia.ceph.com]: reset os type and version on server 2021-06-22T23:15:11.896 INFO:teuthology.lock.ops:Updating gibba030.front.sepia.ceph.com on lock server 2021-06-22T23:15:11.909 INFO:teuthology.lock.ops:Node 'ubuntu@gibba030.front.sepia.ceph.com' reimaging is complete 2021-06-22T23:15:12.112 INFO:teuthology.provision.fog.gibba030:Scheduling deploy of ubuntu 20.04 2021-06-22T23:15:12.121 INFO:teuthology.provision.fog.gibba010:Scheduling deploy of ubuntu 20.04 2021-06-22T23:15:12.183 INFO:teuthology.provision.fog.gibba028:Scheduling deploy of ubuntu 20.04 2021-06-22T23:15:12.479 INFO:teuthology.orchestra.console:Power off gibba030 2021-06-22T23:15:12.590 INFO:teuthology.orchestra.console:Power off gibba010 2021-06-22T23:15:12.623 INFO:teuthology.orchestra.console:Power off for gibba030 completed 2021-06-22T23:15:12.639 INFO:teuthology.orchestra.console:Power off gibba028 2021-06-22T23:15:12.725 INFO:teuthology.orchestra.console:Power on gibba030 2021-06-22T23:15:12.734 INFO:teuthology.orchestra.console:Power off for gibba010 completed 2021-06-22T23:15:12.785 INFO:teuthology.orchestra.console:Power off for gibba028 completed 2021-06-22T23:15:12.835 INFO:teuthology.orchestra.console:Power on gibba010 2021-06-22T23:15:12.886 INFO:teuthology.orchestra.console:Power on gibba028 2021-06-22T23:15:17.090 INFO:teuthology.orchestra.console:Power on for gibba010 completed 2021-06-22T23:15:17.191 INFO:teuthology.provision.fog.gibba010:Waiting for deploy to finish 2021-06-22T23:15:21.092 INFO:teuthology.orchestra.console:Power on for gibba030 completed 2021-06-22T23:15:21.194 INFO:teuthology.provision.fog.gibba030:Waiting for deploy to finish 2021-06-22T23:15:21.250 INFO:teuthology.orchestra.console:Power on for gibba028 completed 2021-06-22T23:15:21.351 INFO:teuthology.provision.fog.gibba028:Waiting for deploy to finish 2021-06-22T23:20:03.180 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-22T23:20:03.557 INFO:teuthology.provision.fog.gibba028:Node is ready 2021-06-22T23:20:03.561 INFO:teuthology.orchestra.run.gibba028.stdout:gibba028.front.sepia.ceph.com 2021-06-22T23:20:03.566 INFO:teuthology.orchestra.run.gibba028.stdout:172.21.2.128 gibba028.front.sepia.ceph.com gibba028 2021-06-22T23:20:03.802 INFO:teuthology.provision.fog.gibba028:Deploy complete! 2021-06-22T23:20:04.385 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-22T23:20:04.889 INFO:teuthology.provision.fog.gibba010:Node is ready 2021-06-22T23:20:04.893 INFO:teuthology.orchestra.run.gibba010.stdout:gibba010.front.sepia.ceph.com 2021-06-22T23:20:04.898 INFO:teuthology.orchestra.run.gibba010.stdout:172.21.2.110 gibba010.front.sepia.ceph.com gibba010 2021-06-22T23:20:05.050 INFO:teuthology.provision.fog.gibba010:Deploy complete! 2021-06-22T23:20:38.997 ERROR:teuthology.orchestra.connection:Error authenticating with gibba030.front.sepia.ceph.com: Authentication failed. 2021-06-22T23:20:40.093 ERROR:teuthology.orchestra.connection:Error authenticating with gibba030.front.sepia.ceph.com: Authentication failed. 2021-06-22T23:20:41.188 ERROR:teuthology.orchestra.connection:Error authenticating with gibba030.front.sepia.ceph.com: Authentication failed. 2021-06-22T23:20:42.282 ERROR:teuthology.orchestra.connection:Error authenticating with gibba030.front.sepia.ceph.com: Authentication failed. 2021-06-22T23:20:43.377 ERROR:teuthology.orchestra.connection:Error authenticating with gibba030.front.sepia.ceph.com: Authentication failed. 2021-06-22T23:20:44.472 ERROR:teuthology.orchestra.connection:Error authenticating with gibba030.front.sepia.ceph.com: Authentication failed. 2021-06-22T23:20:45.526 ERROR:teuthology.orchestra.connection:Error authenticating with gibba030.front.sepia.ceph.com: Authentication failed. 2021-06-22T23:20:46.580 ERROR:teuthology.orchestra.connection:Error authenticating with gibba030.front.sepia.ceph.com: Authentication failed. 2021-06-22T23:20:47.674 ERROR:teuthology.orchestra.connection:Error authenticating with gibba030.front.sepia.ceph.com: Authentication failed. 2021-06-22T23:20:48.768 ERROR:teuthology.orchestra.connection:Error authenticating with gibba030.front.sepia.ceph.com: Authentication failed. 2021-06-22T23:20:54.771 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-06-22T23:20:54.772 ERROR:paramiko.transport:Traceback (most recent call last): 2021-06-22T23:20:54.772 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-06-22T23:20:54.772 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-06-22T23:20:54.773 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-06-22T23:20:54.773 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-06-22T23:20:54.773 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 609, in _read_timeout 2021-06-22T23:20:54.773 ERROR:paramiko.transport: raise EOFError() 2021-06-22T23:20:54.774 ERROR:paramiko.transport:EOFError 2021-06-22T23:20:54.774 ERROR:paramiko.transport: 2021-06-22T23:20:54.774 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-06-22T23:20:54.775 ERROR:paramiko.transport: 2021-06-22T23:20:54.775 ERROR:paramiko.transport:Traceback (most recent call last): 2021-06-22T23:20:54.775 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-06-22T23:20:54.776 ERROR:paramiko.transport: self._check_banner() 2021-06-22T23:20:54.776 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-06-22T23:20:54.776 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-06-22T23:20:54.776 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-06-22T23:20:54.777 ERROR:paramiko.transport: 2021-06-22T23:21:00.833 ERROR:teuthology.orchestra.connection:Error authenticating with gibba030.front.sepia.ceph.com: Authentication failed. 2021-06-22T23:21:01.928 ERROR:teuthology.orchestra.connection:Error authenticating with gibba030.front.sepia.ceph.com: Authentication failed. 2021-06-22T23:21:02.982 ERROR:teuthology.orchestra.connection:Error authenticating with gibba030.front.sepia.ceph.com: Authentication failed. 2021-06-22T23:21:04.078 ERROR:teuthology.orchestra.connection:Error authenticating with gibba030.front.sepia.ceph.com: Authentication failed. 2021-06-22T23:21:05.173 ERROR:teuthology.orchestra.connection:Error authenticating with gibba030.front.sepia.ceph.com: Authentication failed. 2021-06-22T23:21:06.268 ERROR:teuthology.orchestra.connection:Error authenticating with gibba030.front.sepia.ceph.com: Authentication failed. 2021-06-22T23:21:07.362 ERROR:teuthology.orchestra.connection:Error authenticating with gibba030.front.sepia.ceph.com: Authentication failed. 2021-06-22T23:21:08.366 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-06-22T23:21:08.366 ERROR:paramiko.transport:Traceback (most recent call last): 2021-06-22T23:21:08.367 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-06-22T23:21:08.367 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-06-22T23:21:08.367 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-06-22T23:21:08.368 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-06-22T23:21:08.368 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 609, in _read_timeout 2021-06-22T23:21:08.368 ERROR:paramiko.transport: raise EOFError() 2021-06-22T23:21:08.369 ERROR:paramiko.transport:EOFError 2021-06-22T23:21:08.369 ERROR:paramiko.transport: 2021-06-22T23:21:08.370 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-06-22T23:21:08.372 ERROR:paramiko.transport: 2021-06-22T23:21:08.372 ERROR:paramiko.transport:Traceback (most recent call last): 2021-06-22T23:21:08.373 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-06-22T23:21:08.373 ERROR:paramiko.transport: self._check_banner() 2021-06-22T23:21:08.374 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-06-22T23:21:08.374 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-06-22T23:21:08.375 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-06-22T23:21:08.376 ERROR:paramiko.transport: 2021-06-22T23:21:14.474 ERROR:teuthology.orchestra.connection:Error authenticating with gibba030.front.sepia.ceph.com: Authentication failed. 2021-06-22T23:21:15.568 ERROR:teuthology.orchestra.connection:Error authenticating with gibba030.front.sepia.ceph.com: Authentication failed. 2021-06-22T23:21:16.663 ERROR:teuthology.orchestra.connection:Error authenticating with gibba030.front.sepia.ceph.com: Authentication failed. 2021-06-22T23:21:17.757 ERROR:teuthology.orchestra.connection:Error authenticating with gibba030.front.sepia.ceph.com: Authentication failed. 2021-06-22T23:21:18.852 ERROR:teuthology.orchestra.connection:Error authenticating with gibba030.front.sepia.ceph.com: Authentication failed. 2021-06-22T23:21:19.854 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-06-22T23:21:19.854 ERROR:paramiko.transport:Traceback (most recent call last): 2021-06-22T23:21:19.855 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-06-22T23:21:19.855 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-06-22T23:21:19.855 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-06-22T23:21:19.856 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-06-22T23:21:19.856 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 609, in _read_timeout 2021-06-22T23:21:19.856 ERROR:paramiko.transport: raise EOFError() 2021-06-22T23:21:19.856 ERROR:paramiko.transport:EOFError 2021-06-22T23:21:19.857 ERROR:paramiko.transport: 2021-06-22T23:21:19.857 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-06-22T23:21:19.857 ERROR:paramiko.transport: 2021-06-22T23:21:19.858 ERROR:paramiko.transport:Traceback (most recent call last): 2021-06-22T23:21:19.858 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-06-22T23:21:19.858 ERROR:paramiko.transport: self._check_banner() 2021-06-22T23:21:19.859 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-06-22T23:21:19.859 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-06-22T23:21:19.859 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-06-22T23:21:19.860 ERROR:paramiko.transport: 2021-06-22T23:23:46.997 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-22T23:23:47.075 INFO:teuthology.provision.fog.gibba030:Node is ready 2021-06-22T23:23:47.126 INFO:teuthology.orchestra.run.gibba030.stdout:gibba030 2021-06-22T23:23:47.178 INFO:teuthology.orchestra.run.gibba030.stdout:172.21.2.130 gibba030 gibba030 2021-06-22T23:23:47.473 INFO:teuthology.provision.fog.gibba030:Deploy complete! 2021-06-22T23:23:53.675 ERROR:teuthology.misc:write (gibba010.front.sepia.ceph.com): No route to host 2021-06-22T23:23:55.683 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 163, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/lock/ops.py", line 318, in reimage_machines reimaged = do_update_keys(list(reimaged.keys()))[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/lock/ops.py", line 285, in do_update_keys keys_dict = misc.ssh_keyscan(machines, _raise=_raise) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/misc.py", line 1103, in ssh_keyscan while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'ssh_keyscan gibba010.front.sepia.ceph.com' reached maximum tries (5) after waiting for 5 seconds 2021-06-22T23:23:55.726 INFO:teuthology.nuke:Checking targets against current locks 2021-06-22T23:23:55.806 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-22T23:23:55.810 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-22T23:23:55.811 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-22T23:23:55.871 INFO:teuthology.orchestra.console:Power off gibba010 2021-06-22T23:23:55.885 INFO:teuthology.orchestra.console:Power off gibba028 2021-06-22T23:23:55.897 INFO:teuthology.orchestra.console:Power off gibba030 2021-06-22T23:24:04.266 INFO:teuthology.orchestra.console:Power off for gibba028 completed 2021-06-22T23:24:04.367 INFO:teuthology.orchestra.console:Power off for gibba010 completed 2021-06-22T23:24:04.404 INFO:teuthology.lock.ops:unlocked gibba028.front.sepia.ceph.com 2021-06-22T23:24:04.502 INFO:teuthology.lock.ops:unlocked gibba010.front.sepia.ceph.com 2021-06-22T23:24:08.456 INFO:teuthology.orchestra.console:Power off for gibba030 completed 2021-06-22T23:24:08.594 INFO:teuthology.lock.ops:unlocked gibba030.front.sepia.ceph.com 2021-06-22T23:24:08.595 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/bin/teuthology-dispatcher", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-dispatcher')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/scripts/dispatcher.py", line 33, in main teuthology.dispatcher.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/__init__.py", line 61, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 50, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 163, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/lock/ops.py", line 318, in reimage_machines reimaged = do_update_keys(list(reimaged.keys()))[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/lock/ops.py", line 285, in do_update_keys keys_dict = misc.ssh_keyscan(machines, _raise=_raise) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/misc.py", line 1103, in ssh_keyscan while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'ssh_keyscan gibba010.front.sepia.ceph.com' reached maximum tries (5) after waiting for 5 seconds