2021-09-15T03:56:48.645 INFO:root:teuthology version: 1.1.0-df69d6db 2021-09-15T03:56:48.820 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba008.front.sepia.ceph.com' reimaging 2021-09-15T03:56:48.822 INFO:teuthology.lock.ops:Updating [ubuntu@gibba008.front.sepia.ceph.com]: reset os type and version on server 2021-09-15T03:56:48.822 INFO:teuthology.lock.ops:Updating gibba008.front.sepia.ceph.com on lock server 2021-09-15T03:56:48.838 INFO:teuthology.lock.ops:Node 'ubuntu@gibba008.front.sepia.ceph.com' reimaging is complete 2021-09-15T03:56:48.838 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba010.front.sepia.ceph.com' reimaging 2021-09-15T03:56:48.838 INFO:teuthology.lock.ops:Updating [ubuntu@gibba010.front.sepia.ceph.com]: reset os type and version on server 2021-09-15T03:56:48.839 INFO:teuthology.lock.ops:Updating gibba010.front.sepia.ceph.com on lock server 2021-09-15T03:56:48.853 INFO:teuthology.lock.ops:Node 'ubuntu@gibba010.front.sepia.ceph.com' reimaging is complete 2021-09-15T03:56:48.854 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba020.front.sepia.ceph.com' reimaging 2021-09-15T03:56:48.854 INFO:teuthology.lock.ops:Updating [ubuntu@gibba020.front.sepia.ceph.com]: reset os type and version on server 2021-09-15T03:56:48.855 INFO:teuthology.lock.ops:Updating gibba020.front.sepia.ceph.com on lock server 2021-09-15T03:56:48.869 INFO:teuthology.lock.ops:Node 'ubuntu@gibba020.front.sepia.ceph.com' reimaging is complete 2021-09-15T03:56:48.870 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba027.front.sepia.ceph.com' reimaging 2021-09-15T03:56:48.870 INFO:teuthology.lock.ops:Updating [ubuntu@gibba027.front.sepia.ceph.com]: reset os type and version on server 2021-09-15T03:56:48.870 INFO:teuthology.lock.ops:Updating gibba027.front.sepia.ceph.com on lock server 2021-09-15T03:56:48.884 INFO:teuthology.lock.ops:Node 'ubuntu@gibba027.front.sepia.ceph.com' reimaging is complete 2021-09-15T03:56:48.885 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba036.front.sepia.ceph.com' reimaging 2021-09-15T03:56:48.885 INFO:teuthology.lock.ops:Updating [ubuntu@gibba036.front.sepia.ceph.com]: reset os type and version on server 2021-09-15T03:56:48.885 INFO:teuthology.lock.ops:Updating gibba036.front.sepia.ceph.com on lock server 2021-09-15T03:56:48.902 INFO:teuthology.lock.ops:Node 'ubuntu@gibba036.front.sepia.ceph.com' reimaging is complete 2021-09-15T03:56:49.062 INFO:teuthology.provision.fog.gibba010:Scheduling deploy of ubuntu 18.04 2021-09-15T03:56:49.066 INFO:teuthology.provision.fog.gibba008:Scheduling deploy of ubuntu 18.04 2021-09-15T03:56:49.077 INFO:teuthology.provision.fog.gibba020:Scheduling deploy of ubuntu 18.04 2021-09-15T03:56:49.103 INFO:teuthology.provision.fog.gibba036:Scheduling deploy of ubuntu 18.04 2021-09-15T03:56:49.117 INFO:teuthology.provision.fog.gibba027:Scheduling deploy of ubuntu 18.04 2021-09-15T03:56:49.393 INFO:teuthology.orchestra.console:Power off gibba008 2021-09-15T03:56:49.409 INFO:teuthology.orchestra.console:Power off gibba010 2021-09-15T03:56:49.420 INFO:teuthology.orchestra.console:Power off gibba036 2021-09-15T03:56:49.438 INFO:teuthology.orchestra.console:Power off gibba020 2021-09-15T03:56:49.486 INFO:teuthology.orchestra.console:Power off gibba027 2021-09-15T03:56:57.891 INFO:teuthology.orchestra.console:Power off for gibba036 completed 2021-09-15T03:56:57.992 INFO:teuthology.orchestra.console:Power on gibba036 2021-09-15T03:57:01.914 INFO:teuthology.orchestra.console:Power off for gibba008 completed 2021-09-15T03:57:01.923 INFO:teuthology.orchestra.console:Power off for gibba010 completed 2021-09-15T03:57:01.960 INFO:teuthology.orchestra.console:Power off for gibba027 completed 2021-09-15T03:57:02.005 INFO:teuthology.orchestra.console:Power off for gibba020 completed 2021-09-15T03:57:02.016 INFO:teuthology.orchestra.console:Power on gibba008 2021-09-15T03:57:02.024 INFO:teuthology.orchestra.console:Power on gibba010 2021-09-15T03:57:02.062 INFO:teuthology.orchestra.console:Power on gibba027 2021-09-15T03:57:02.106 INFO:teuthology.orchestra.console:Power on gibba020 2021-09-15T03:57:06.370 INFO:teuthology.orchestra.console:Power on for gibba036 completed 2021-09-15T03:57:06.471 INFO:teuthology.provision.fog.gibba036:Waiting for deploy to finish 2021-09-15T03:57:10.398 INFO:teuthology.orchestra.console:Power on for gibba008 completed 2021-09-15T03:57:10.433 INFO:teuthology.orchestra.console:Power on for gibba027 completed 2021-09-15T03:57:10.469 INFO:teuthology.orchestra.console:Power on for gibba010 completed 2021-09-15T03:57:10.474 INFO:teuthology.orchestra.console:Power on for gibba020 completed 2021-09-15T03:57:10.500 INFO:teuthology.provision.fog.gibba008:Waiting for deploy to finish 2021-09-15T03:57:10.534 INFO:teuthology.provision.fog.gibba027:Waiting for deploy to finish 2021-09-15T03:57:10.570 INFO:teuthology.provision.fog.gibba010:Waiting for deploy to finish 2021-09-15T03:57:10.575 INFO:teuthology.provision.fog.gibba020:Waiting for deploy to finish 2021-09-15T04:01:33.866 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-09-15T04:01:37.865 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-09-15T04:01:39.888 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-09-15T04:01:43.881 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-09-15T04:01:45.898 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-09-15T04:01:51.909 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-09-15T04:01:53.505 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-09-15T04:01:53.697 INFO:teuthology.provision.fog.gibba036:Node is ready 2021-09-15T04:01:53.739 INFO:teuthology.orchestra.run.gibba036.stdout:gibba036.front.sepia.ceph.com 2021-09-15T04:01:53.784 INFO:teuthology.orchestra.run.gibba036.stdout:172.21.2.136 gibba036.front.sepia.ceph.com gibba036 2021-09-15T04:01:54.058 INFO:teuthology.provision.fog.gibba036:Deploy complete! 2021-09-15T04:01:55.972 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-09-15T04:01:56.165 INFO:teuthology.provision.fog.gibba020:Node is ready 2021-09-15T04:01:56.208 INFO:teuthology.orchestra.run.gibba020.stdout:gibba020.front.sepia.ceph.com 2021-09-15T04:01:56.252 INFO:teuthology.orchestra.run.gibba020.stdout:172.21.2.120 gibba020.front.sepia.ceph.com gibba020 2021-09-15T04:01:56.539 INFO:teuthology.provision.fog.gibba020:Deploy complete! 2021-09-15T04:01:57.595 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-09-15T04:01:57.781 INFO:teuthology.provision.fog.gibba010:Node is ready 2021-09-15T04:01:57.826 INFO:teuthology.orchestra.run.gibba010.stdout:gibba010.front.sepia.ceph.com 2021-09-15T04:01:57.870 INFO:teuthology.orchestra.run.gibba010.stdout:172.21.2.110 gibba010.front.sepia.ceph.com gibba010 2021-09-15T04:01:57.982 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-09-15T04:01:58.132 INFO:teuthology.provision.fog.gibba010:Deploy complete! 2021-09-15T04:01:58.229 INFO:teuthology.provision.fog.gibba027:Node is ready 2021-09-15T04:01:58.278 INFO:teuthology.orchestra.run.gibba027.stdout:gibba027.front.sepia.ceph.com 2021-09-15T04:01:58.331 INFO:teuthology.orchestra.run.gibba027.stdout:172.21.2.127 gibba027.front.sepia.ceph.com gibba027 2021-09-15T04:01:58.591 INFO:teuthology.provision.fog.gibba027:Deploy complete! 2021-09-15T04:14:56.558 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/dispatcher/supervisor.py", line 163, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/lock/ops.py", line 317, in reimage_machines log.info("Node '%s' reimaging is complete", machine) File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/provision/__init__.py", line 39, in reimage return obj.create() File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/provision/fog.py", line 260, in _wait_for_ready while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds 2021-09-15T04:14:56.613 INFO:teuthology.nuke:Checking targets against current locks 2021-09-15T04:14:56.748 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-15T04:14:56.755 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-15T04:14:56.756 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-15T04:14:56.758 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-15T04:14:56.760 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-15T04:14:56.820 INFO:teuthology.orchestra.console:Power off gibba027 2021-09-15T04:14:56.841 INFO:teuthology.orchestra.console:Power off gibba036 2021-09-15T04:14:56.856 INFO:teuthology.orchestra.console:Power off gibba020 2021-09-15T04:14:56.871 INFO:teuthology.orchestra.console:Power off gibba008 2021-09-15T04:14:56.893 INFO:teuthology.orchestra.console:Power off gibba010 2021-09-15T04:15:05.257 INFO:teuthology.orchestra.console:Power off for gibba036 completed 2021-09-15T04:15:05.269 INFO:teuthology.orchestra.console:Power off for gibba020 completed 2021-09-15T04:15:05.276 INFO:teuthology.orchestra.console:Power off for gibba010 completed 2021-09-15T04:15:05.282 INFO:teuthology.orchestra.console:Power off for gibba027 completed 2021-09-15T04:15:05.407 INFO:teuthology.lock.ops:unlocked gibba036.front.sepia.ceph.com 2021-09-15T04:15:05.410 ERROR:teuthology.lock.ops:failed to unlock gibba020.front.sepia.ceph.com. reason: 500 2021-09-15T04:15:05.424 INFO:teuthology.lock.ops:unlocked gibba027.front.sepia.ceph.com 2021-09-15T04:15:05.428 INFO:teuthology.lock.ops:unlocked gibba010.front.sepia.ceph.com 2021-09-15T04:15:09.378 INFO:teuthology.orchestra.console:Power off for gibba008 completed 2021-09-15T04:15:09.520 INFO:teuthology.lock.ops:unlocked gibba008.front.sepia.ceph.com 2021-09-15T04:15:09.521 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/virtualenv/bin/teuthology-dispatcher", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-dispatcher')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/scripts/dispatcher.py", line 33, in main teuthology.dispatcher.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/dispatcher/__init__.py", line 61, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/dispatcher/supervisor.py", line 50, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/dispatcher/supervisor.py", line 163, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/lock/ops.py", line 317, in reimage_machines log.info("Node '%s' reimaging is complete", machine) File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/provision/__init__.py", line 39, in reimage return obj.create() File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/provision/fog.py", line 260, in _wait_for_ready while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds