2022-10-25T06:07:14.484 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9 2022-10-25T06:07:14.721 INFO:teuthology.lock.ops:Start node 'smithi040.front.sepia.ceph.com' reimaging 2022-10-25T06:07:14.723 INFO:teuthology.lock.ops:Updating [smithi040.front.sepia.ceph.com]: reset os type and version on server 2022-10-25T06:07:14.723 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2022-10-25T06:07:14.747 INFO:teuthology.lock.ops:Node 'smithi040.front.sepia.ceph.com' reimaging is complete 2022-10-25T06:07:14.747 INFO:teuthology.lock.ops:Start node 'smithi071.front.sepia.ceph.com' reimaging 2022-10-25T06:07:14.748 INFO:teuthology.lock.ops:Updating [smithi071.front.sepia.ceph.com]: reset os type and version on server 2022-10-25T06:07:14.748 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2022-10-25T06:07:14.770 INFO:teuthology.lock.ops:Node 'smithi071.front.sepia.ceph.com' reimaging is complete 2022-10-25T06:07:14.770 INFO:teuthology.lock.ops:Start node 'smithi074.front.sepia.ceph.com' reimaging 2022-10-25T06:07:14.770 INFO:teuthology.lock.ops:Updating [smithi074.front.sepia.ceph.com]: reset os type and version on server 2022-10-25T06:07:14.771 INFO:teuthology.lock.ops:Updating smithi074.front.sepia.ceph.com on lock server 2022-10-25T06:07:14.790 INFO:teuthology.lock.ops:Node 'smithi074.front.sepia.ceph.com' reimaging is complete 2022-10-25T06:07:14.790 INFO:teuthology.lock.ops:Start node 'smithi164.front.sepia.ceph.com' reimaging 2022-10-25T06:07:14.791 INFO:teuthology.lock.ops:Updating [smithi164.front.sepia.ceph.com]: reset os type and version on server 2022-10-25T06:07:14.791 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server 2022-10-25T06:07:14.818 INFO:teuthology.lock.ops:Node 'smithi164.front.sepia.ceph.com' reimaging is complete 2022-10-25T06:07:14.948 INFO:teuthology.provision.fog.smithi040:Scheduling deploy of ubuntu 20.04 2022-10-25T06:07:14.990 INFO:teuthology.provision.fog.smithi071:Scheduling deploy of ubuntu 20.04 2022-10-25T06:07:15.002 INFO:teuthology.provision.fog.smithi074:Scheduling deploy of ubuntu 20.04 2022-10-25T06:07:15.042 INFO:teuthology.provision.fog.smithi164:Scheduling deploy of ubuntu 20.04 2022-10-25T06:07:15.311 INFO:teuthology.orchestra.console:Power off smithi071 2022-10-25T06:07:15.334 INFO:teuthology.orchestra.console:Power off smithi040 2022-10-25T06:07:15.376 INFO:teuthology.orchestra.console:Power off smithi074 2022-10-25T06:07:15.417 INFO:teuthology.orchestra.console:Power off smithi164 2022-10-25T06:07:15.488 INFO:teuthology.orchestra.console:Power off for smithi040 completed 2022-10-25T06:07:15.589 INFO:teuthology.orchestra.console:Power on smithi040 2022-10-25T06:07:23.752 INFO:teuthology.orchestra.console:Power off for smithi074 completed 2022-10-25T06:07:23.854 INFO:teuthology.orchestra.console:Power on smithi074 2022-10-25T06:07:23.955 INFO:teuthology.orchestra.console:Power on for smithi040 completed 2022-10-25T06:07:24.058 INFO:teuthology.provision.fog.smithi040:Waiting for deploy to finish 2022-10-25T06:07:27.827 INFO:teuthology.orchestra.console:Power off for smithi071 completed 2022-10-25T06:07:27.914 INFO:teuthology.orchestra.console:Power off for smithi164 completed 2022-10-25T06:07:27.930 INFO:teuthology.orchestra.console:Power on smithi071 2022-10-25T06:07:28.015 INFO:teuthology.orchestra.console:Power on smithi164 2022-10-25T06:07:32.196 INFO:teuthology.orchestra.console:Power on for smithi071 completed 2022-10-25T06:07:32.219 INFO:teuthology.orchestra.console:Power on for smithi074 completed 2022-10-25T06:07:32.278 INFO:teuthology.orchestra.console:Power on for smithi164 completed 2022-10-25T06:07:32.298 INFO:teuthology.provision.fog.smithi071:Waiting for deploy to finish 2022-10-25T06:07:32.320 INFO:teuthology.provision.fog.smithi074:Waiting for deploy to finish 2022-10-25T06:07:32.380 INFO:teuthology.provision.fog.smithi164:Waiting for deploy to finish 2022-10-25T06:11:42.908 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-10-25T06:11:43.322 INFO:teuthology.provision.fog.smithi040:Node is ready 2022-10-25T06:11:43.342 INFO:teuthology.orchestra.run.smithi040.stdout:smithi040.front.sepia.ceph.com 2022-10-25T06:11:43.399 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40 smithi040.front.sepia.ceph.com smithi040 2022-10-25T06:11:43.559 INFO:teuthology.provision.fog.smithi040:Deploy complete! 2022-10-25T06:12:27.045 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-10-25T06:12:27.553 INFO:teuthology.provision.fog.smithi071:Node is ready 2022-10-25T06:12:27.561 INFO:teuthology.orchestra.run.smithi071.stdout:smithi071.front.sepia.ceph.com 2022-10-25T06:12:27.567 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.71 smithi071.front.sepia.ceph.com smithi071 2022-10-25T06:12:27.798 INFO:teuthology.provision.fog.smithi071:Deploy complete! 2022-10-25T06:12:29.092 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-10-25T06:12:29.533 INFO:teuthology.provision.fog.smithi074:Node is ready 2022-10-25T06:12:29.540 INFO:teuthology.orchestra.run.smithi074.stdout:smithi074.front.sepia.ceph.com 2022-10-25T06:12:29.547 INFO:teuthology.orchestra.run.smithi074.stdout:172.21.15.74 smithi074.front.sepia.ceph.com smithi074 2022-10-25T06:12:29.727 INFO:teuthology.provision.fog.smithi074:Deploy complete! 2022-10-25T06:12:54.198 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-10-25T06:12:54.628 INFO:teuthology.provision.fog.smithi164:Node is ready 2022-10-25T06:12:54.636 INFO:teuthology.orchestra.run.smithi164.stdout:smithi164.front.sepia.ceph.com 2022-10-25T06:12:54.644 INFO:teuthology.orchestra.run.smithi164.stdout:172.21.15.164 smithi164.front.sepia.ceph.com smithi164 2022-10-25T06:12:54.866 INFO:teuthology.provision.fog.smithi164:Deploy complete! 2022-10-25T06:12:54.991 ERROR:teuthology.misc:write (smithi040.front.sepia.ceph.com): Connection refused 2022-10-25T06:12:54.993 ERROR:teuthology.misc:write (smithi040.front.sepia.ceph.com): Connection refused 2022-10-25T06:12:54.993 ERROR:teuthology.misc:write (smithi040.front.sepia.ceph.com): Connection refused 2022-10-25T06:12:56.002 ERROR:teuthology.misc:write (smithi040.front.sepia.ceph.com): Connection refused 2022-10-25T06:12:56.003 ERROR:teuthology.misc:write (smithi040.front.sepia.ceph.com): Connection refused 2022-10-25T06:12:56.003 ERROR:teuthology.misc:write (smithi040.front.sepia.ceph.com): Connection refused 2022-10-25T06:12:57.014 ERROR:teuthology.misc:write (smithi040.front.sepia.ceph.com): Connection refused 2022-10-25T06:12:57.015 ERROR:teuthology.misc:write (smithi040.front.sepia.ceph.com): Connection refused 2022-10-25T06:12:57.016 ERROR:teuthology.misc:write (smithi040.front.sepia.ceph.com): Connection refused 2022-10-25T06:12:58.032 ERROR:teuthology.misc:write (smithi040.front.sepia.ceph.com): Connection refused 2022-10-25T06:12:58.033 ERROR:teuthology.misc:write (smithi040.front.sepia.ceph.com): Connection refused 2022-10-25T06:12:58.033 ERROR:teuthology.misc:write (smithi040.front.sepia.ceph.com): Connection refused 2022-10-25T06:13:00.042 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/dispatcher/supervisor.py", line 209, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/lock/ops.py", line 325, in reimage_machines reimaged = do_update_keys(list(reimaged.keys()))[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/lock/ops.py", line 288, in do_update_keys keys_dict = misc.ssh_keyscan(machines, _raise=_raise) File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/misc.py", line 1106, in ssh_keyscan while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'ssh_keyscan smithi040.front.sepia.ceph.com' reached maximum tries (5) after waiting for 5 seconds 2022-10-25T06:13:00.131 INFO:teuthology.nuke:Checking targets against current locks 2022-10-25T06:13:00.195 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-10-25T06:13:00.226 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-10-25T06:13:00.294 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-10-25T06:13:00.311 INFO:teuthology.orchestra.console:Power off smithi040 2022-10-25T06:13:00.334 INFO:teuthology.orchestra.console:Power off smithi071 2022-10-25T06:13:00.359 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-10-25T06:13:00.426 INFO:teuthology.orchestra.console:Power off smithi074 2022-10-25T06:13:00.472 INFO:teuthology.orchestra.console:Power off smithi164 2022-10-25T06:13:08.710 INFO:teuthology.orchestra.console:Power off for smithi040 completed 2022-10-25T06:13:08.810 INFO:teuthology.orchestra.console:Power off for smithi074 completed 2022-10-25T06:13:08.840 INFO:teuthology.orchestra.console:Power off for smithi164 completed 2022-10-25T06:13:08.871 INFO:teuthology.lock.ops:unlocked: smithi040.front.sepia.ceph.com 2022-10-25T06:13:08.969 INFO:teuthology.lock.ops:unlocked: smithi074.front.sepia.ceph.com 2022-10-25T06:13:09.009 INFO:teuthology.lock.ops:unlocked: smithi164.front.sepia.ceph.com 2022-10-25T06:13:12.840 INFO:teuthology.orchestra.console:Power off for smithi071 completed 2022-10-25T06:13:13.009 INFO:teuthology.lock.ops:unlocked: smithi071.front.sepia.ceph.com 2022-10-25T06:13:13.461 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/virtualenv/bin/teuthology-dispatcher", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-dispatcher')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/dispatcher/__init__.py", line 61, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/dispatcher/supervisor.py", line 52, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/dispatcher/supervisor.py", line 209, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/lock/ops.py", line 325, in reimage_machines reimaged = do_update_keys(list(reimaged.keys()))[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/lock/ops.py", line 288, in do_update_keys keys_dict = misc.ssh_keyscan(machines, _raise=_raise) File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/misc.py", line 1106, in ssh_keyscan while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'ssh_keyscan smithi040.front.sepia.ceph.com' reached maximum tries (5) after waiting for 5 seconds