2022-09-27T11:29:29.679 INFO:root:teuthology version: 1.1.1.dev424+ga773b061 2022-09-27T11:29:29.850 INFO:teuthology.lock.ops:Start node 'smithi003.front.sepia.ceph.com' reimaging 2022-09-27T11:29:29.851 INFO:teuthology.lock.ops:Updating [smithi003.front.sepia.ceph.com]: reset os type and version on server 2022-09-27T11:29:29.851 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2022-09-27T11:29:29.874 INFO:teuthology.lock.ops:Node 'smithi003.front.sepia.ceph.com' reimaging is complete 2022-09-27T11:29:29.874 INFO:teuthology.lock.ops:Start node 'smithi008.front.sepia.ceph.com' reimaging 2022-09-27T11:29:29.875 INFO:teuthology.lock.ops:Updating [smithi008.front.sepia.ceph.com]: reset os type and version on server 2022-09-27T11:29:29.876 INFO:teuthology.lock.ops:Updating smithi008.front.sepia.ceph.com on lock server 2022-09-27T11:29:29.897 INFO:teuthology.lock.ops:Node 'smithi008.front.sepia.ceph.com' reimaging is complete 2022-09-27T11:29:29.897 INFO:teuthology.lock.ops:Start node 'smithi043.front.sepia.ceph.com' reimaging 2022-09-27T11:29:29.898 INFO:teuthology.lock.ops:Updating [smithi043.front.sepia.ceph.com]: reset os type and version on server 2022-09-27T11:29:29.898 INFO:teuthology.lock.ops:Updating smithi043.front.sepia.ceph.com on lock server 2022-09-27T11:29:29.916 INFO:teuthology.lock.ops:Node 'smithi043.front.sepia.ceph.com' reimaging is complete 2022-09-27T11:29:30.100 INFO:teuthology.provision.fog.smithi003:Scheduling deploy of centos 8.stream 2022-09-27T11:29:30.113 INFO:teuthology.provision.fog.smithi008:Scheduling deploy of centos 8.stream 2022-09-27T11:29:30.151 INFO:teuthology.provision.fog.smithi043:Scheduling deploy of centos 8.stream 2022-09-27T11:29:30.423 INFO:teuthology.orchestra.console:Power off smithi003 2022-09-27T11:29:30.440 INFO:teuthology.orchestra.console:Power off smithi008 2022-09-27T11:29:30.504 INFO:teuthology.orchestra.console:Power off smithi043 2022-09-27T11:29:38.807 INFO:teuthology.orchestra.console:Power off for smithi003 completed 2022-09-27T11:29:38.812 INFO:teuthology.orchestra.console:Power off for smithi008 completed 2022-09-27T11:29:38.878 INFO:teuthology.orchestra.console:Power off for smithi043 completed 2022-09-27T11:29:38.909 INFO:teuthology.orchestra.console:Power on smithi003 2022-09-27T11:29:38.918 INFO:teuthology.orchestra.console:Power on smithi008 2022-09-27T11:29:38.980 INFO:teuthology.orchestra.console:Power on smithi043 2022-09-27T11:29:43.234 INFO:teuthology.orchestra.console:Power on for smithi043 completed 2022-09-27T11:29:43.335 INFO:teuthology.provision.fog.smithi043:Waiting for deploy to finish 2022-09-27T11:29:47.274 INFO:teuthology.orchestra.console:Power on for smithi003 completed 2022-09-27T11:29:47.323 INFO:teuthology.orchestra.console:Power on for smithi008 completed 2022-09-27T11:29:47.376 INFO:teuthology.provision.fog.smithi003:Waiting for deploy to finish 2022-09-27T11:29:47.424 INFO:teuthology.provision.fog.smithi008:Waiting for deploy to finish 2022-09-27T11:33:31.564 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-09-27T11:35:25.777 ERROR:teuthology.orchestra.connection:Error authenticating with smithi008.front.sepia.ceph.com: Authentication failed. 2022-09-27T11:35:26.886 ERROR:teuthology.orchestra.connection:Error authenticating with smithi008.front.sepia.ceph.com: Authentication failed. 2022-09-27T11:35:27.994 ERROR:teuthology.orchestra.connection:Error authenticating with smithi008.front.sepia.ceph.com: Authentication failed. 2022-09-27T11:35:29.104 ERROR:teuthology.orchestra.connection:Error authenticating with smithi008.front.sepia.ceph.com: Authentication failed. 2022-09-27T11:35:30.208 ERROR:teuthology.orchestra.connection:Error authenticating with smithi008.front.sepia.ceph.com: Authentication failed. 2022-09-27T11:35:31.287 ERROR:teuthology.orchestra.connection:Error authenticating with smithi008.front.sepia.ceph.com: Authentication failed. 2022-09-27T11:35:32.397 ERROR:teuthology.orchestra.connection:Error authenticating with smithi008.front.sepia.ceph.com: Authentication failed. 2022-09-27T11:35:33.479 ERROR:teuthology.orchestra.connection:Error authenticating with smithi008.front.sepia.ceph.com: Authentication failed. 2022-09-27T11:35:34.609 ERROR:teuthology.orchestra.connection:Error authenticating with smithi008.front.sepia.ceph.com: Authentication failed. 2022-09-27T11:35:35.678 ERROR:teuthology.orchestra.connection:Error authenticating with smithi008.front.sepia.ceph.com: Authentication failed. 2022-09-27T11:35:41.762 ERROR:teuthology.orchestra.connection:Error authenticating with smithi008.front.sepia.ceph.com: Authentication failed. 2022-09-27T11:35:42.886 ERROR:teuthology.orchestra.connection:Error authenticating with smithi008.front.sepia.ceph.com: Authentication failed. 2022-09-27T11:35:43.889 ERROR:paramiko.transport:Exception (client): Error reading SSH protocol banner 2022-09-27T11:35:43.890 ERROR:paramiko.transport:Traceback (most recent call last): 2022-09-27T11:35:43.891 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2271, in _check_banner 2022-09-27T11:35:43.891 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2022-09-27T11:35:43.891 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2022-09-27T11:35:43.892 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2022-09-27T11:35:43.892 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 609, in _read_timeout 2022-09-27T11:35:43.892 ERROR:paramiko.transport: raise EOFError() 2022-09-27T11:35:43.893 ERROR:paramiko.transport:EOFError 2022-09-27T11:35:43.893 ERROR:paramiko.transport: 2022-09-27T11:35:43.893 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2022-09-27T11:35:43.894 ERROR:paramiko.transport: 2022-09-27T11:35:43.894 ERROR:paramiko.transport:Traceback (most recent call last): 2022-09-27T11:35:43.894 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2094, in run 2022-09-27T11:35:43.895 ERROR:paramiko.transport: self._check_banner() 2022-09-27T11:35:43.895 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2276, in _check_banner 2022-09-27T11:35:43.895 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2022-09-27T11:35:43.895 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2022-09-27T11:35:43.896 ERROR:paramiko.transport: 2022-09-27T11:35:50.024 ERROR:teuthology.orchestra.connection:Error authenticating with smithi008.front.sepia.ceph.com: Authentication failed. 2022-09-27T11:35:51.097 ERROR:teuthology.orchestra.connection:Error authenticating with smithi008.front.sepia.ceph.com: Authentication failed. 2022-09-27T11:35:52.210 ERROR:teuthology.orchestra.connection:Error authenticating with smithi008.front.sepia.ceph.com: Authentication failed. 2022-09-27T11:35:53.213 ERROR:paramiko.transport:Exception (client): Error reading SSH protocol banner 2022-09-27T11:35:53.214 ERROR:paramiko.transport:Traceback (most recent call last): 2022-09-27T11:35:53.215 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2271, in _check_banner 2022-09-27T11:35:53.216 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2022-09-27T11:35:53.216 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2022-09-27T11:35:53.217 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2022-09-27T11:35:53.218 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 609, in _read_timeout 2022-09-27T11:35:53.218 ERROR:paramiko.transport: raise EOFError() 2022-09-27T11:35:53.219 ERROR:paramiko.transport:EOFError 2022-09-27T11:35:53.219 ERROR:paramiko.transport: 2022-09-27T11:35:53.220 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2022-09-27T11:35:53.220 ERROR:paramiko.transport: 2022-09-27T11:35:53.221 ERROR:paramiko.transport:Traceback (most recent call last): 2022-09-27T11:35:53.222 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2094, in run 2022-09-27T11:35:53.222 ERROR:paramiko.transport: self._check_banner() 2022-09-27T11:35:53.223 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2276, in _check_banner 2022-09-27T11:35:53.223 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2022-09-27T11:35:53.224 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2022-09-27T11:35:53.226 ERROR:paramiko.transport: 2022-09-27T11:35:56.923 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2022-09-27T11:35:56.927 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/dispatcher/supervisor.py", line 209, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/lock/ops.py", line 324, in reimage_machines log.info("Node '%s' reimaging is complete", machine) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/provision/__init__.py", line 39, in reimage return obj.create() File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/provision/fog.py", line 286, in _wait_for_ready self.remote.run(args=cmd, timeout=600) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/orchestra/run.py", line 175, in _raise_for_status node=self.hostname) teuthology.exceptions.ConnectionLostError: SSH connection to smithi043 was lost: "while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done" 2022-09-27T11:35:57.015 INFO:teuthology.nuke:Checking targets against current locks 2022-09-27T11:35:57.050 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/dispatcher/supervisor.py", line 209, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/lock/ops.py", line 324, in reimage_machines log.info("Node '%s' reimaging is complete", machine) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/provision/__init__.py", line 39, in reimage return obj.create() File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/provision/fog.py", line 286, in _wait_for_ready self.remote.run(args=cmd, timeout=600) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/orchestra/run.py", line 175, in _raise_for_status node=self.hostname) teuthology.exceptions.ConnectionLostError: SSH connection to smithi043 was lost: "while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/virtualenv/bin/teuthology-dispatcher", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-dispatcher')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/dispatcher/__init__.py", line 61, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/dispatcher/supervisor.py", line 52, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/dispatcher/supervisor.py", line 214, in reimage nuke(ctx, True) File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/nuke/__init__.py", line 246, in nuke if ctx.name and ctx.name not in status.get('description', ""): TypeError: argument of type 'NoneType' is not iterable