2021-06-22T22:54:27.519 INFO:root:teuthology version: 1.1.0-f359b10d 2021-06-22T22:54:27.608 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba043.front.sepia.ceph.com' reimaging 2021-06-22T22:54:27.609 INFO:teuthology.lock.ops:Updating [ubuntu@gibba043.front.sepia.ceph.com]: reset os type and version on server 2021-06-22T22:54:27.610 INFO:teuthology.lock.ops:Updating gibba043.front.sepia.ceph.com on lock server 2021-06-22T22:54:27.623 INFO:teuthology.lock.ops:Node 'ubuntu@gibba043.front.sepia.ceph.com' reimaging is complete 2021-06-22T22:54:27.624 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba045.front.sepia.ceph.com' reimaging 2021-06-22T22:54:27.624 INFO:teuthology.lock.ops:Updating [ubuntu@gibba045.front.sepia.ceph.com]: reset os type and version on server 2021-06-22T22:54:27.624 INFO:teuthology.lock.ops:Updating gibba045.front.sepia.ceph.com on lock server 2021-06-22T22:54:27.640 INFO:teuthology.lock.ops:Node 'ubuntu@gibba045.front.sepia.ceph.com' reimaging is complete 2021-06-22T22:54:27.867 INFO:teuthology.provision.fog.gibba045:Scheduling deploy of centos 8.3 2021-06-22T22:54:27.877 INFO:teuthology.provision.fog.gibba043:Scheduling deploy of centos 8.3 2021-06-22T22:54:28.194 INFO:teuthology.orchestra.console:Power off gibba045 2021-06-22T22:54:28.287 INFO:teuthology.orchestra.console:Power off gibba043 2021-06-22T22:54:28.347 INFO:teuthology.orchestra.console:Power off for gibba045 completed 2021-06-22T22:54:28.449 INFO:teuthology.orchestra.console:Power on gibba045 2021-06-22T22:54:28.456 INFO:teuthology.orchestra.console:Power off for gibba043 completed 2021-06-22T22:54:28.557 INFO:teuthology.orchestra.console:Power on gibba043 2021-06-22T22:54:36.820 INFO:teuthology.orchestra.console:Power on for gibba045 completed 2021-06-22T22:54:36.922 INFO:teuthology.provision.fog.gibba045:Waiting for deploy to finish 2021-06-22T22:54:36.929 INFO:teuthology.orchestra.console:Power on for gibba043 completed 2021-06-22T22:54:37.030 INFO:teuthology.provision.fog.gibba043:Waiting for deploy to finish 2021-06-22T22:58:08.986 ERROR:teuthology.orchestra.connection:Error authenticating with gibba045.front.sepia.ceph.com: Authentication failed. 2021-06-22T22:58:10.083 ERROR:teuthology.orchestra.connection:Error authenticating with gibba045.front.sepia.ceph.com: Authentication failed. 2021-06-22T22:58:11.178 ERROR:teuthology.orchestra.connection:Error authenticating with gibba045.front.sepia.ceph.com: Authentication failed. 2021-06-22T22:58:12.272 ERROR:teuthology.orchestra.connection:Error authenticating with gibba045.front.sepia.ceph.com: Authentication failed. 2021-06-22T22:58:13.367 ERROR:teuthology.orchestra.connection:Error authenticating with gibba045.front.sepia.ceph.com: Authentication failed. 2021-06-22T22:58:14.462 ERROR:teuthology.orchestra.connection:Error authenticating with gibba045.front.sepia.ceph.com: Authentication failed. 2021-06-22T22:58:15.516 ERROR:teuthology.orchestra.connection:Error authenticating with gibba045.front.sepia.ceph.com: Authentication failed. 2021-06-22T22:58:16.569 ERROR:teuthology.orchestra.connection:Error authenticating with gibba045.front.sepia.ceph.com: Authentication failed. 2021-06-22T22:58:17.664 ERROR:teuthology.orchestra.connection:Error authenticating with gibba045.front.sepia.ceph.com: Authentication failed. 2021-06-22T22:58:18.717 ERROR:teuthology.orchestra.connection:Error authenticating with gibba045.front.sepia.ceph.com: Authentication failed. 2021-06-22T22:58:24.773 ERROR:teuthology.orchestra.connection:Error authenticating with gibba045.front.sepia.ceph.com: Authentication failed. 2021-06-22T22:58:25.867 ERROR:teuthology.orchestra.connection:Error authenticating with gibba045.front.sepia.ceph.com: Authentication failed. 2021-06-22T22:58:26.870 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-06-22T22:58:26.871 ERROR:paramiko.transport:Traceback (most recent call last): 2021-06-22T22:58:26.871 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-06-22T22:58:26.872 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-06-22T22:58:26.872 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-06-22T22:58:26.872 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-06-22T22:58:26.873 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 609, in _read_timeout 2021-06-22T22:58:26.873 ERROR:paramiko.transport: raise EOFError() 2021-06-22T22:58:26.873 ERROR:paramiko.transport:EOFError 2021-06-22T22:58:26.873 ERROR:paramiko.transport: 2021-06-22T22:58:26.874 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-06-22T22:58:26.874 ERROR:paramiko.transport: 2021-06-22T22:58:26.874 ERROR:paramiko.transport:Traceback (most recent call last): 2021-06-22T22:58:26.874 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-06-22T22:58:26.875 ERROR:paramiko.transport: self._check_banner() 2021-06-22T22:58:26.875 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-06-22T22:58:26.875 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-06-22T22:58:26.876 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-06-22T22:58:26.876 ERROR:paramiko.transport: 2021-06-22T22:58:32.932 ERROR:teuthology.orchestra.connection:Error authenticating with gibba045.front.sepia.ceph.com: Authentication failed. 2021-06-22T22:58:34.027 ERROR:teuthology.orchestra.connection:Error authenticating with gibba045.front.sepia.ceph.com: Authentication failed. 2021-06-22T22:58:35.029 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-06-22T22:58:35.030 ERROR:paramiko.transport:Traceback (most recent call last): 2021-06-22T22:58:35.030 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-06-22T22:58:35.031 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-06-22T22:58:35.031 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-06-22T22:58:35.031 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-06-22T22:58:35.031 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 609, in _read_timeout 2021-06-22T22:58:35.032 ERROR:paramiko.transport: raise EOFError() 2021-06-22T22:58:35.032 ERROR:paramiko.transport:EOFError 2021-06-22T22:58:35.032 ERROR:paramiko.transport: 2021-06-22T22:58:35.033 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-06-22T22:58:35.033 ERROR:paramiko.transport: 2021-06-22T22:58:35.033 ERROR:paramiko.transport:Traceback (most recent call last): 2021-06-22T22:58:35.033 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-06-22T22:58:35.034 ERROR:paramiko.transport: self._check_banner() 2021-06-22T22:58:35.034 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-06-22T22:58:35.034 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-06-22T22:58:35.035 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-06-22T22:58:35.035 ERROR:paramiko.transport: 2021-06-22T22:58:37.196 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-22T22:58:41.131 ERROR:teuthology.orchestra.connection:Error authenticating with gibba045.front.sepia.ceph.com: Authentication failed. 2021-06-22T22:58:42.225 ERROR:teuthology.orchestra.connection:Error authenticating with gibba045.front.sepia.ceph.com: Authentication failed. 2021-06-22T22:58:43.227 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-06-22T22:58:43.228 ERROR:paramiko.transport:Traceback (most recent call last): 2021-06-22T22:58:43.228 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-06-22T22:58:43.228 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-06-22T22:58:43.228 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-06-22T22:58:43.229 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-06-22T22:58:43.229 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 609, in _read_timeout 2021-06-22T22:58:43.229 ERROR:paramiko.transport: raise EOFError() 2021-06-22T22:58:43.229 ERROR:paramiko.transport:EOFError 2021-06-22T22:58:43.230 ERROR:paramiko.transport: 2021-06-22T22:58:43.230 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-06-22T22:58:43.230 ERROR:paramiko.transport: 2021-06-22T22:58:43.231 ERROR:paramiko.transport:Traceback (most recent call last): 2021-06-22T22:58:43.231 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-06-22T22:58:43.231 ERROR:paramiko.transport: self._check_banner() 2021-06-22T22:58:43.232 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-06-22T22:58:43.232 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-06-22T22:58:43.232 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-06-22T22:58:43.232 ERROR:paramiko.transport: 2021-06-22T22:58:49.329 ERROR:teuthology.orchestra.connection:Error authenticating with gibba045.front.sepia.ceph.com: Authentication failed. 2021-06-22T22:58:50.422 ERROR:teuthology.orchestra.connection:Error authenticating with gibba045.front.sepia.ceph.com: Authentication failed. 2021-06-22T22:58:51.517 ERROR:teuthology.orchestra.connection:Error authenticating with gibba045.front.sepia.ceph.com: Authentication failed. 2021-06-22T22:58:52.316 INFO:teuthology.provision.fog.gibba043:Node is ready 2021-06-22T22:58:52.328 INFO:teuthology.orchestra.run.gibba043.stdout:gibba043 2021-06-22T22:58:52.380 INFO:teuthology.orchestra.run.gibba043.stdout:172.21.2.143 gibba043 gibba043 2021-06-22T22:58:52.612 ERROR:teuthology.orchestra.connection:Error authenticating with gibba045.front.sepia.ceph.com: Authentication failed. 2021-06-22T22:58:52.676 INFO:teuthology.provision.fog.gibba043:Deploy complete! 2021-06-22T22:58:53.706 ERROR:teuthology.orchestra.connection:Error authenticating with gibba045.front.sepia.ceph.com: Authentication failed. 2021-06-22T23:01:07.025 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-22T23:01:07.296 INFO:teuthology.provision.fog.gibba045:Node is ready 2021-06-22T23:01:07.307 INFO:teuthology.orchestra.run.gibba045.stdout:gibba045.front.sepia.ceph.com 2021-06-22T23:01:07.359 INFO:teuthology.orchestra.run.gibba045.stdout:172.21.2.145 gibba045.front.sepia.ceph.com gibba045 2021-06-22T23:01:07.644 INFO:teuthology.provision.fog.gibba045:Deploy complete! 2021-06-22T23:01:08.651 ERROR:teuthology.misc:write (gibba043.front.sepia.ceph.com): No route to host 2021-06-22T23:01:11.723 ERROR:teuthology.misc:write (gibba043.front.sepia.ceph.com): No route to host 2021-06-22T23:01:14.795 ERROR:teuthology.misc:write (gibba043.front.sepia.ceph.com): No route to host 2021-06-22T23:01:14.796 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 163, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/lock/ops.py", line 318, in reimage_machines reimaged = do_update_keys(list(reimaged.keys()))[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/lock/ops.py", line 285, in do_update_keys keys_dict = misc.ssh_keyscan(machines, _raise=_raise) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/misc.py", line 1103, in ssh_keyscan while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'ssh_keyscan gibba043.front.sepia.ceph.com' reached maximum tries (5) after waiting for 5 seconds 2021-06-22T23:01:14.845 INFO:teuthology.nuke:Checking targets against current locks 2021-06-22T23:01:14.930 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-22T23:01:14.933 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-22T23:01:14.985 INFO:teuthology.orchestra.console:Power off gibba045 2021-06-22T23:01:15.000 INFO:teuthology.orchestra.console:Power off gibba043 2021-06-22T23:01:19.343 INFO:teuthology.orchestra.console:Power off for gibba043 completed 2021-06-22T23:01:19.480 INFO:teuthology.lock.ops:unlocked gibba043.front.sepia.ceph.com 2021-06-22T23:01:23.360 INFO:teuthology.orchestra.console:Power off for gibba045 completed 2021-06-22T23:01:23.499 INFO:teuthology.lock.ops:unlocked gibba045.front.sepia.ceph.com 2021-06-22T23:01:23.500 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/bin/teuthology-dispatcher", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-dispatcher')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/scripts/dispatcher.py", line 33, in main teuthology.dispatcher.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/__init__.py", line 61, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 50, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 163, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/lock/ops.py", line 318, in reimage_machines reimaged = do_update_keys(list(reimaged.keys()))[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/lock/ops.py", line 285, in do_update_keys keys_dict = misc.ssh_keyscan(machines, _raise=_raise) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/misc.py", line 1103, in ssh_keyscan while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'ssh_keyscan gibba043.front.sepia.ceph.com' reached maximum tries (5) after waiting for 5 seconds