2021-04-01T19:33:07.890 INFO:root:teuthology version: 1.1.0-85d61eae 2021-04-01T19:33:08.046 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba008.front.sepia.ceph.com' reimaging 2021-04-01T19:33:08.047 INFO:teuthology.lock.ops:Updating [ubuntu@gibba008.front.sepia.ceph.com]: reset os type and version on server 2021-04-01T19:33:08.047 INFO:teuthology.lock.ops:Updating gibba008.front.sepia.ceph.com on lock server 2021-04-01T19:33:08.061 INFO:teuthology.lock.ops:Node 'ubuntu@gibba008.front.sepia.ceph.com' reimaging is complete 2021-04-01T19:33:08.061 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba023.front.sepia.ceph.com' reimaging 2021-04-01T19:33:08.061 INFO:teuthology.lock.ops:Updating [ubuntu@gibba023.front.sepia.ceph.com]: reset os type and version on server 2021-04-01T19:33:08.062 INFO:teuthology.lock.ops:Updating gibba023.front.sepia.ceph.com on lock server 2021-04-01T19:33:08.077 INFO:teuthology.lock.ops:Node 'ubuntu@gibba023.front.sepia.ceph.com' reimaging is complete 2021-04-01T19:33:08.077 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba029.front.sepia.ceph.com' reimaging 2021-04-01T19:33:08.077 INFO:teuthology.lock.ops:Updating [ubuntu@gibba029.front.sepia.ceph.com]: reset os type and version on server 2021-04-01T19:33:08.078 INFO:teuthology.lock.ops:Updating gibba029.front.sepia.ceph.com on lock server 2021-04-01T19:33:08.091 INFO:teuthology.lock.ops:Node 'ubuntu@gibba029.front.sepia.ceph.com' reimaging is complete 2021-04-01T19:33:08.092 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba040.front.sepia.ceph.com' reimaging 2021-04-01T19:33:08.092 INFO:teuthology.lock.ops:Updating [ubuntu@gibba040.front.sepia.ceph.com]: reset os type and version on server 2021-04-01T19:33:08.092 INFO:teuthology.lock.ops:Updating gibba040.front.sepia.ceph.com on lock server 2021-04-01T19:33:08.106 INFO:teuthology.lock.ops:Node 'ubuntu@gibba040.front.sepia.ceph.com' reimaging is complete 2021-04-01T19:33:08.106 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba042.front.sepia.ceph.com' reimaging 2021-04-01T19:33:08.106 INFO:teuthology.lock.ops:Updating [ubuntu@gibba042.front.sepia.ceph.com]: reset os type and version on server 2021-04-01T19:33:08.107 INFO:teuthology.lock.ops:Updating gibba042.front.sepia.ceph.com on lock server 2021-04-01T19:33:08.120 INFO:teuthology.lock.ops:Node 'ubuntu@gibba042.front.sepia.ceph.com' reimaging is complete 2021-04-01T19:33:08.303 INFO:teuthology.provision.fog.gibba008:Scheduling deploy of centos 7.8 2021-04-01T19:33:08.310 INFO:teuthology.provision.fog.gibba023:Scheduling deploy of centos 7.8 2021-04-01T19:33:08.321 INFO:teuthology.provision.fog.gibba040:Scheduling deploy of centos 7.8 2021-04-01T19:33:08.335 INFO:teuthology.provision.fog.gibba029:Scheduling deploy of centos 7.8 2021-04-01T19:33:08.342 INFO:teuthology.provision.fog.gibba042:Scheduling deploy of centos 7.8 2021-04-01T19:33:08.639 INFO:teuthology.orchestra.console:Power off gibba040 2021-04-01T19:33:08.654 INFO:teuthology.orchestra.console:Power off gibba023 2021-04-01T19:33:08.688 INFO:teuthology.orchestra.console:Power off gibba042 2021-04-01T19:33:08.701 INFO:teuthology.orchestra.console:Power off gibba008 2021-04-01T19:33:08.727 INFO:teuthology.orchestra.console:Power off gibba029 2021-04-01T19:33:08.811 INFO:teuthology.orchestra.console:Power off for gibba040 completed 2021-04-01T19:33:08.817 INFO:teuthology.orchestra.console:Power off for gibba023 completed 2021-04-01T19:33:08.835 INFO:teuthology.orchestra.console:Power off for gibba042 completed 2021-04-01T19:33:08.857 INFO:teuthology.orchestra.console:Power off for gibba008 completed 2021-04-01T19:33:08.871 INFO:teuthology.orchestra.console:Power off for gibba029 completed 2021-04-01T19:33:08.912 INFO:teuthology.orchestra.console:Power on gibba040 2021-04-01T19:33:08.919 INFO:teuthology.orchestra.console:Power on gibba023 2021-04-01T19:33:08.940 INFO:teuthology.orchestra.console:Power on gibba042 2021-04-01T19:33:08.959 INFO:teuthology.orchestra.console:Power on gibba008 2021-04-01T19:33:08.971 INFO:teuthology.orchestra.console:Power on gibba029 2021-04-01T19:33:13.198 INFO:teuthology.orchestra.console:Power on for gibba042 completed 2021-04-01T19:33:13.299 INFO:teuthology.provision.fog.gibba042:Waiting for deploy to finish 2021-04-01T19:33:17.279 INFO:teuthology.orchestra.console:Power on for gibba040 completed 2021-04-01T19:33:17.293 INFO:teuthology.orchestra.console:Power on for gibba023 completed 2021-04-01T19:33:17.321 INFO:teuthology.orchestra.console:Power on for gibba008 completed 2021-04-01T19:33:17.332 INFO:teuthology.orchestra.console:Power on for gibba029 completed 2021-04-01T19:33:17.381 INFO:teuthology.provision.fog.gibba040:Waiting for deploy to finish 2021-04-01T19:33:17.394 INFO:teuthology.provision.fog.gibba023:Waiting for deploy to finish 2021-04-01T19:33:17.422 INFO:teuthology.provision.fog.gibba008:Waiting for deploy to finish 2021-04-01T19:33:17.433 INFO:teuthology.provision.fog.gibba029:Waiting for deploy to finish 2021-04-01T19:36:48.349 ERROR:teuthology.orchestra.connection:Error authenticating with gibba008.front.sepia.ceph.com: Authentication failed. 2021-04-01T19:36:49.406 ERROR:teuthology.orchestra.connection:Error authenticating with gibba008.front.sepia.ceph.com: Authentication failed. 2021-04-01T19:36:50.500 ERROR:teuthology.orchestra.connection:Error authenticating with gibba008.front.sepia.ceph.com: Authentication failed. 2021-04-01T19:36:51.594 ERROR:teuthology.orchestra.connection:Error authenticating with gibba008.front.sepia.ceph.com: Authentication failed. 2021-04-01T19:36:52.688 ERROR:teuthology.orchestra.connection:Error authenticating with gibba008.front.sepia.ceph.com: Authentication failed. 2021-04-01T19:36:53.782 ERROR:teuthology.orchestra.connection:Error authenticating with gibba008.front.sepia.ceph.com: Authentication failed. 2021-04-01T19:36:54.836 ERROR:teuthology.orchestra.connection:Error authenticating with gibba008.front.sepia.ceph.com: Authentication failed. 2021-04-01T19:36:55.931 ERROR:teuthology.orchestra.connection:Error authenticating with gibba008.front.sepia.ceph.com: Authentication failed. 2021-04-01T19:36:56.985 ERROR:teuthology.orchestra.connection:Error authenticating with gibba008.front.sepia.ceph.com: Authentication failed. 2021-04-01T19:36:58.039 ERROR:teuthology.orchestra.connection:Error authenticating with gibba008.front.sepia.ceph.com: Authentication failed. 2021-04-01T19:37:04.094 ERROR:teuthology.orchestra.connection:Error authenticating with gibba008.front.sepia.ceph.com: Authentication failed. 2021-04-01T19:37:05.103 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-04-01T19:37:05.104 ERROR:paramiko.transport:Traceback (most recent call last): 2021-04-01T19:37:05.105 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-04-01T19:37:05.105 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-04-01T19:37:05.105 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-04-01T19:37:05.106 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-04-01T19:37:05.106 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 609, in _read_timeout 2021-04-01T19:37:05.107 ERROR:paramiko.transport: raise EOFError() 2021-04-01T19:37:05.107 ERROR:paramiko.transport:EOFError 2021-04-01T19:37:05.107 ERROR:paramiko.transport: 2021-04-01T19:37:05.108 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-04-01T19:37:05.108 ERROR:paramiko.transport: 2021-04-01T19:37:05.109 ERROR:paramiko.transport:Traceback (most recent call last): 2021-04-01T19:37:05.109 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-04-01T19:37:05.109 ERROR:paramiko.transport: self._check_banner() 2021-04-01T19:37:05.110 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-04-01T19:37:05.110 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-04-01T19:37:05.111 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-04-01T19:37:05.111 ERROR:paramiko.transport: 2021-04-01T19:37:11.205 ERROR:teuthology.orchestra.connection:Error authenticating with gibba008.front.sepia.ceph.com: Authentication failed. 2021-04-01T19:37:12.299 ERROR:teuthology.orchestra.connection:Error authenticating with gibba008.front.sepia.ceph.com: Authentication failed. 2021-04-01T19:37:13.302 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-04-01T19:37:13.303 ERROR:paramiko.transport:Traceback (most recent call last): 2021-04-01T19:37:13.303 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-04-01T19:37:13.303 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-04-01T19:37:13.304 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-04-01T19:37:13.304 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-04-01T19:37:13.304 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 609, in _read_timeout 2021-04-01T19:37:13.304 ERROR:paramiko.transport: raise EOFError() 2021-04-01T19:37:13.305 ERROR:paramiko.transport:EOFError 2021-04-01T19:37:13.305 ERROR:paramiko.transport: 2021-04-01T19:37:13.305 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-04-01T19:37:13.306 ERROR:paramiko.transport: 2021-04-01T19:37:13.306 ERROR:paramiko.transport:Traceback (most recent call last): 2021-04-01T19:37:13.306 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-04-01T19:37:13.307 ERROR:paramiko.transport: self._check_banner() 2021-04-01T19:37:13.307 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-04-01T19:37:13.307 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-04-01T19:37:13.307 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-04-01T19:37:13.308 ERROR:paramiko.transport: 2021-04-01T19:37:19.402 ERROR:teuthology.orchestra.connection:Error authenticating with gibba008.front.sepia.ceph.com: Authentication failed. 2021-04-01T19:37:20.413 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-04-01T19:37:20.414 ERROR:paramiko.transport:Traceback (most recent call last): 2021-04-01T19:37:20.414 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-04-01T19:37:20.414 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-04-01T19:37:20.414 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-04-01T19:37:20.415 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-04-01T19:37:20.415 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 609, in _read_timeout 2021-04-01T19:37:20.415 ERROR:paramiko.transport: raise EOFError() 2021-04-01T19:37:20.416 ERROR:paramiko.transport:EOFError 2021-04-01T19:37:20.416 ERROR:paramiko.transport: 2021-04-01T19:37:20.417 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-04-01T19:37:20.417 ERROR:paramiko.transport: 2021-04-01T19:37:20.418 ERROR:paramiko.transport:Traceback (most recent call last): 2021-04-01T19:37:20.418 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-04-01T19:37:20.418 ERROR:paramiko.transport: self._check_banner() 2021-04-01T19:37:20.418 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-04-01T19:37:20.419 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-04-01T19:37:20.419 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-04-01T19:37:20.419 ERROR:paramiko.transport: 2021-04-01T19:37:20.553 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-01T19:37:26.514 ERROR:teuthology.orchestra.connection:Error authenticating with gibba008.front.sepia.ceph.com: Authentication failed. 2021-04-01T19:37:27.610 ERROR:teuthology.orchestra.connection:Error authenticating with gibba008.front.sepia.ceph.com: Authentication failed. 2021-04-01T19:37:28.704 ERROR:teuthology.orchestra.connection:Error authenticating with gibba008.front.sepia.ceph.com: Authentication failed. 2021-04-01T19:37:29.798 ERROR:teuthology.orchestra.connection:Error authenticating with gibba008.front.sepia.ceph.com: Authentication failed. 2021-04-01T19:37:30.895 ERROR:teuthology.orchestra.connection:Error authenticating with gibba008.front.sepia.ceph.com: Authentication failed. 2021-04-01T19:37:31.493 INFO:teuthology.provision.fog.gibba040:Node is ready 2021-04-01T19:37:31.916 INFO:teuthology.orchestra.run.gibba040.stdout:gibba004 2021-04-01T19:37:31.989 ERROR:teuthology.orchestra.connection:Error authenticating with gibba008.front.sepia.ceph.com: Authentication failed. 2021-04-01T19:37:32.316 INFO:teuthology.orchestra.run.gibba040.stdout:172.21.2.104 gibba004.front.sepia.ceph.com gibba004 2021-04-01T19:37:33.083 ERROR:teuthology.orchestra.connection:Error authenticating with gibba008.front.sepia.ceph.com: Authentication failed. 2021-04-01T19:37:34.085 ERROR:paramiko.transport:Exception: Error reading SSH protocol banner 2021-04-01T19:37:34.085 ERROR:paramiko.transport:Traceback (most recent call last): 2021-04-01T19:37:34.086 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2211, in _check_banner 2021-04-01T19:37:34.086 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2021-04-01T19:37:34.086 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2021-04-01T19:37:34.087 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2021-04-01T19:37:34.087 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 609, in _read_timeout 2021-04-01T19:37:34.087 ERROR:paramiko.transport: raise EOFError() 2021-04-01T19:37:34.088 ERROR:paramiko.transport:EOFError 2021-04-01T19:37:34.088 ERROR:paramiko.transport: 2021-04-01T19:37:34.088 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2021-04-01T19:37:34.089 ERROR:paramiko.transport: 2021-04-01T19:37:34.089 ERROR:paramiko.transport:Traceback (most recent call last): 2021-04-01T19:37:34.089 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2039, in run 2021-04-01T19:37:34.090 ERROR:paramiko.transport: self._check_banner() 2021-04-01T19:37:34.090 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2216, in _check_banner 2021-04-01T19:37:34.090 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2021-04-01T19:37:34.090 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2021-04-01T19:37:34.091 ERROR:paramiko.transport: 2021-04-01T19:37:34.216 INFO:teuthology.provision.fog.gibba040:Deploy complete! 2021-04-01T19:40:01.215 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-01T19:40:01.720 INFO:teuthology.provision.fog.gibba008:Node is ready 2021-04-01T19:40:02.055 INFO:teuthology.orchestra.run.gibba008.stdout:gibba008 2021-04-01T19:40:02.392 INFO:teuthology.orchestra.run.gibba008.stdout:172.21.2.108 gibba008 gibba008 2021-04-01T19:40:04.132 INFO:teuthology.provision.fog.gibba008:Deploy complete! 2021-04-01T19:48:02.956 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/dispatcher/supervisor.py", line 163, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/lock/ops.py", line 317, in reimage_machines log.info("Node '%s' reimaging is complete", machine) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/provision/__init__.py", line 39, in reimage return obj.create() File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/provision/fog.py", line 86, in create self.wait_for_deploy_task(task_id) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/provision/fog.py", line 244, in wait_for_deploy_task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (60) after waiting for 900 seconds 2021-04-01T19:48:03.021 INFO:teuthology.nuke:Checking targets against current locks 2021-04-01T19:48:03.288 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-01T19:48:03.300 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-01T19:48:03.312 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-01T19:48:03.326 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-01T19:48:03.341 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-01T19:48:03.483 INFO:teuthology.orchestra.console:Power off gibba040 2021-04-01T19:48:03.497 INFO:teuthology.orchestra.console:Power off gibba029 2021-04-01T19:48:03.509 INFO:teuthology.orchestra.console:Power off gibba023 2021-04-01T19:48:03.521 INFO:teuthology.orchestra.console:Power off gibba008 2021-04-01T19:48:03.539 INFO:teuthology.orchestra.console:Power off gibba042 2021-04-01T19:48:03.675 INFO:teuthology.orchestra.console:Power off for gibba029 completed 2021-04-01T19:48:03.676 INFO:teuthology.orchestra.console:Power off for gibba008 completed 2021-04-01T19:48:03.682 INFO:teuthology.orchestra.console:Power off for gibba023 completed 2021-04-01T19:48:03.690 INFO:teuthology.orchestra.console:Power off for gibba042 completed 2021-04-01T19:48:03.847 INFO:teuthology.lock.ops:unlocked gibba029.front.sepia.ceph.com 2021-04-01T19:48:03.863 INFO:teuthology.lock.ops:unlocked gibba008.front.sepia.ceph.com 2021-04-01T19:48:03.880 INFO:teuthology.lock.ops:unlocked gibba023.front.sepia.ceph.com 2021-04-01T19:48:03.896 INFO:teuthology.lock.ops:unlocked gibba042.front.sepia.ceph.com 2021-04-01T19:48:11.886 INFO:teuthology.orchestra.console:Power off for gibba040 completed 2021-04-01T19:48:12.027 INFO:teuthology.lock.ops:unlocked gibba040.front.sepia.ceph.com 2021-04-01T19:48:12.028 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/bin/teuthology-dispatcher", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-dispatcher')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/scripts/dispatcher.py", line 33, in main teuthology.dispatcher.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/dispatcher/__init__.py", line 61, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/dispatcher/supervisor.py", line 50, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/dispatcher/supervisor.py", line 163, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/lock/ops.py", line 317, in reimage_machines log.info("Node '%s' reimaging is complete", machine) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/provision/__init__.py", line 39, in reimage return obj.create() File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/provision/fog.py", line 86, in create self.wait_for_deploy_task(task_id) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/provision/fog.py", line 244, in wait_for_deploy_task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (60) after waiting for 900 seconds