2021-09-28T21:06:22.923 INFO:root:teuthology version: 1.1.0-f08b273e 2021-09-28T21:06:23.086 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi046.front.sepia.ceph.com' reimaging 2021-09-28T21:06:23.087 INFO:teuthology.lock.ops:Updating [ubuntu@smithi046.front.sepia.ceph.com]: reset os type and version on server 2021-09-28T21:06:23.088 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2021-09-28T21:06:23.107 INFO:teuthology.lock.ops:Node 'ubuntu@smithi046.front.sepia.ceph.com' reimaging is complete 2021-09-28T21:06:23.107 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi068.front.sepia.ceph.com' reimaging 2021-09-28T21:06:23.107 INFO:teuthology.lock.ops:Updating [ubuntu@smithi068.front.sepia.ceph.com]: reset os type and version on server 2021-09-28T21:06:23.108 INFO:teuthology.lock.ops:Updating smithi068.front.sepia.ceph.com on lock server 2021-09-28T21:06:23.127 INFO:teuthology.lock.ops:Node 'ubuntu@smithi068.front.sepia.ceph.com' reimaging is complete 2021-09-28T21:06:23.128 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi107.front.sepia.ceph.com' reimaging 2021-09-28T21:06:23.128 INFO:teuthology.lock.ops:Updating [ubuntu@smithi107.front.sepia.ceph.com]: reset os type and version on server 2021-09-28T21:06:23.128 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2021-09-28T21:06:23.146 INFO:teuthology.lock.ops:Node 'ubuntu@smithi107.front.sepia.ceph.com' reimaging is complete 2021-09-28T21:06:23.323 INFO:teuthology.provision.fog.smithi046:Scheduling deploy of ubuntu 18.04 2021-09-28T21:06:23.337 INFO:teuthology.provision.fog.smithi068:Scheduling deploy of ubuntu 18.04 2021-09-28T21:06:23.359 INFO:teuthology.provision.fog.smithi107:Scheduling deploy of ubuntu 18.04 2021-09-28T21:06:23.718 INFO:teuthology.orchestra.console:Power off smithi068 2021-09-28T21:06:23.784 INFO:teuthology.orchestra.console:Power off smithi046 2021-09-28T21:06:23.850 INFO:teuthology.orchestra.console:Power off smithi107 2021-09-28T21:06:23.994 INFO:teuthology.orchestra.console:Power off for smithi107 completed 2021-09-28T21:06:24.096 INFO:teuthology.orchestra.console:Power on smithi107 2021-09-28T21:06:28.363 INFO:teuthology.orchestra.console:Power on for smithi107 completed 2021-09-28T21:06:28.466 INFO:teuthology.provision.fog.smithi107:Waiting for deploy to finish 2021-09-28T21:06:32.154 INFO:teuthology.orchestra.console:Power off for smithi046 completed 2021-09-28T21:06:32.255 INFO:teuthology.orchestra.console:Power on smithi046 2021-09-28T21:06:36.206 INFO:teuthology.orchestra.console:Power off for smithi068 completed 2021-09-28T21:06:36.309 INFO:teuthology.orchestra.console:Power on smithi068 2021-09-28T21:06:40.623 INFO:teuthology.orchestra.console:Power on for smithi046 completed 2021-09-28T21:06:40.724 INFO:teuthology.provision.fog.smithi046:Waiting for deploy to finish 2021-09-28T21:06:44.684 INFO:teuthology.orchestra.console:Power on for smithi068 completed 2021-09-28T21:06:44.785 INFO:teuthology.provision.fog.smithi068:Waiting for deploy to finish 2021-09-28T21:11:02.354 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-09-28T21:11:08.374 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-09-28T21:11:14.390 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-09-28T21:11:20.405 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-09-28T21:11:26.513 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-09-28T21:11:26.808 INFO:teuthology.provision.fog.smithi046:Node is ready 2021-09-28T21:11:26.855 INFO:teuthology.orchestra.run.smithi046.stdout:smithi046.front.sepia.ceph.com 2021-09-28T21:11:26.904 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46 smithi046.front.sepia.ceph.com smithi046 2021-09-28T21:11:27.218 INFO:teuthology.provision.fog.smithi046:Deploy complete! 2021-09-28T21:11:42.128 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-09-28T21:11:42.243 INFO:teuthology.provision.fog.smithi107:Node is ready 2021-09-28T21:11:42.291 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107 2021-09-28T21:11:42.339 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107 smithi107 smithi107 2021-09-28T21:11:42.619 INFO:teuthology.provision.fog.smithi107:Deploy complete! 2021-09-28T21:11:55.846 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-09-28T21:11:56.108 INFO:teuthology.provision.fog.smithi068:Node is ready 2021-09-28T21:11:56.157 INFO:teuthology.orchestra.run.smithi068.stdout:smithi068.front.sepia.ceph.com 2021-09-28T21:11:56.211 INFO:teuthology.orchestra.run.smithi068.stdout:172.21.15.68 smithi068.front.sepia.ceph.com smithi068 2021-09-28T21:11:56.529 INFO:teuthology.provision.fog.smithi068:Deploy complete! 2021-09-28T21:11:56.695 ERROR:teuthology.misc:write (smithi107.front.sepia.ceph.com): Connection refused 2021-09-28T21:12:04.732 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/teuthology/dispatcher/supervisor.py", line 163, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/teuthology/lock/ops.py", line 316, in reimage_machines reimaged = do_update_keys(list(reimaged.keys()))[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/teuthology/lock/ops.py", line 283, in do_update_keys keys_dict = misc.ssh_keyscan(machines, _raise=_raise) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/teuthology/misc.py", line 1103, in ssh_keyscan while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'ssh_keyscan smithi107.front.sepia.ceph.com' reached maximum tries (5) after waiting for 5 seconds 2021-09-28T21:12:04.794 INFO:teuthology.nuke:Checking targets against current locks 2021-09-28T21:12:04.934 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-28T21:12:04.938 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-28T21:12:04.940 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-28T21:12:04.962 ERROR:teuthology.nuke:Could not nuke {'ubuntu@smithi107.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV+l3NiuIbO9LearcaPNvpLKyYr0GDcrkrCAcJog9lM0JF9uXD6YEiAPlosCQCC0DTEQM4lsZ7dkeamOICrtpsDqhSdIAGk0p2BlqPrYVJePbl1xFkzVZA35z8e72c/vwGBMtt/fkJ3HOKPysnBhB5MBuNf4Quc9dAmWJVyxfAse+CUkHw46ODNLdZWKOLftrIGoWI9KyrD1WoUZ15W5Cj1ZwnD/fS2pvHe1t8JENa1/lAIwPW9E6Y576ry0UGHL8mMFrigorXjtk3zlYUUDhuYfTkDp5xDM+u1mXd3JzRObNFxKj4uKdWVY6vq+K9xUXOHLocHy4OC0HWmM6u1AjL'} Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/teuthology/nuke/__init__.py", line 292, in nuke_one nuke_helper(ctx, should_unlock, keep_logs, should_reboot) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/teuthology/nuke/__init__.py", line 315, in nuke_helper check_lock.check_lock(ctx, None, check_up=False) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/teuthology/task/internal/check_lock.py", line 29, in check_lock 'machine {name} is not locked'.format(name=machine) AssertionError: machine ubuntu@smithi107.front.sepia.ceph.com is not locked 2021-09-28T21:12:05.027 INFO:teuthology.orchestra.console:Power off smithi046 2021-09-28T21:12:05.042 INFO:teuthology.orchestra.console:Power off smithi068 2021-09-28T21:12:13.405 INFO:teuthology.orchestra.console:Power off for smithi046 completed 2021-09-28T21:12:13.568 INFO:teuthology.lock.ops:unlocked: smithi046.front.sepia.ceph.com 2021-09-28T21:12:17.523 INFO:teuthology.orchestra.console:Power off for smithi068 completed 2021-09-28T21:12:17.683 INFO:teuthology.lock.ops:unlocked: smithi068.front.sepia.ceph.com 2021-09-28T21:12:17.685 ERROR:teuthology.nuke:Could not nuke the following targets: targets: ubuntu@smithi107.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV+l3NiuIbO9LearcaPNvpLKyYr0GDcrkrCAcJog9lM0JF9uXD6YEiAPlosCQCC0DTEQM4lsZ7dkeamOICrtpsDqhSdIAGk0p2BlqPrYVJePbl1xFkzVZA35z8e72c/vwGBMtt/fkJ3HOKPysnBhB5MBuNf4Quc9dAmWJVyxfAse+CUkHw46ODNLdZWKOLftrIGoWI9KyrD1WoUZ15W5Cj1ZwnD/fS2pvHe1t8JENa1/lAIwPW9E6Y576ry0UGHL8mMFrigorXjtk3zlYUUDhuYfTkDp5xDM+u1mXd3JzRObNFxKj4uKdWVY6vq+K9xUXOHLocHy4OC0HWmM6u1AjL 2021-09-28T21:12:17.686 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/bin/teuthology-dispatcher", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-dispatcher')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/scripts/dispatcher.py", line 33, in main teuthology.dispatcher.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/teuthology/dispatcher/__init__.py", line 61, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/teuthology/dispatcher/supervisor.py", line 50, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/teuthology/dispatcher/supervisor.py", line 163, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/teuthology/lock/ops.py", line 316, in reimage_machines reimaged = do_update_keys(list(reimaged.keys()))[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/teuthology/lock/ops.py", line 283, in do_update_keys keys_dict = misc.ssh_keyscan(machines, _raise=_raise) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/teuthology/misc.py", line 1103, in ssh_keyscan while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'ssh_keyscan smithi107.front.sepia.ceph.com' reached maximum tries (5) after waiting for 5 seconds