2023-03-12T18:28:08.273 INFO:root:teuthology version: 0.0.1.dev57+g5c9057f 2023-03-12T18:28:08.472 INFO:teuthology.lock.ops:Start node 'smithi002.front.sepia.ceph.com' reimaging 2023-03-12T18:28:08.473 INFO:teuthology.lock.ops:Updating [smithi002.front.sepia.ceph.com]: reset os type and version on server 2023-03-12T18:28:08.473 INFO:teuthology.lock.ops:Updating smithi002.front.sepia.ceph.com on lock server 2023-03-12T18:28:08.489 INFO:teuthology.lock.ops:Node 'smithi002.front.sepia.ceph.com' reimaging is complete 2023-03-12T18:28:08.490 INFO:teuthology.lock.ops:Start node 'smithi102.front.sepia.ceph.com' reimaging 2023-03-12T18:28:08.490 INFO:teuthology.lock.ops:Updating [smithi102.front.sepia.ceph.com]: reset os type and version on server 2023-03-12T18:28:08.490 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server 2023-03-12T18:28:08.514 INFO:teuthology.lock.ops:Node 'smithi102.front.sepia.ceph.com' reimaging is complete 2023-03-12T18:28:08.515 INFO:teuthology.lock.ops:Start node 'smithi122.front.sepia.ceph.com' reimaging 2023-03-12T18:28:08.515 INFO:teuthology.lock.ops:Updating [smithi122.front.sepia.ceph.com]: reset os type and version on server 2023-03-12T18:28:08.515 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2023-03-12T18:28:08.531 INFO:teuthology.lock.ops:Node 'smithi122.front.sepia.ceph.com' reimaging is complete 2023-03-12T18:28:08.531 INFO:teuthology.lock.ops:Start node 'smithi178.front.sepia.ceph.com' reimaging 2023-03-12T18:28:08.531 INFO:teuthology.lock.ops:Updating [smithi178.front.sepia.ceph.com]: reset os type and version on server 2023-03-12T18:28:08.532 INFO:teuthology.lock.ops:Updating smithi178.front.sepia.ceph.com on lock server 2023-03-12T18:28:08.547 INFO:teuthology.lock.ops:Node 'smithi178.front.sepia.ceph.com' reimaging is complete 2023-03-12T18:28:08.643 INFO:teuthology.provision.fog.smithi002:Scheduling deploy of rhel 8.6 2023-03-12T18:28:08.672 INFO:teuthology.provision.fog.smithi122:Scheduling deploy of rhel 8.6 2023-03-12T18:28:08.676 INFO:teuthology.provision.fog.smithi102:Scheduling deploy of rhel 8.6 2023-03-12T18:28:08.690 INFO:teuthology.provision.fog.smithi178:Scheduling deploy of rhel 8.6 2023-03-12T18:28:09.038 INFO:teuthology.orchestra.console:Power off smithi122 2023-03-12T18:28:09.055 INFO:teuthology.orchestra.console:Power off smithi178 2023-03-12T18:28:09.066 INFO:teuthology.orchestra.console:Power off smithi102 2023-03-12T18:28:09.079 INFO:teuthology.orchestra.console:Power off smithi002 2023-03-12T18:28:09.218 INFO:teuthology.orchestra.console:Power off for smithi102 completed 2023-03-12T18:28:09.319 INFO:teuthology.orchestra.console:Power on smithi102 2023-03-12T18:28:13.572 INFO:teuthology.orchestra.console:Power on for smithi102 completed 2023-03-12T18:28:13.673 INFO:teuthology.provision.fog.smithi102:Waiting for deploy to finish 2023-03-12T18:28:21.559 INFO:teuthology.orchestra.console:Power off for smithi178 completed 2023-03-12T18:28:21.565 INFO:teuthology.orchestra.console:Power off for smithi122 completed 2023-03-12T18:28:21.575 INFO:teuthology.orchestra.console:Power off for smithi002 completed 2023-03-12T18:28:21.661 INFO:teuthology.orchestra.console:Power on smithi178 2023-03-12T18:28:21.669 INFO:teuthology.orchestra.console:Power on smithi122 2023-03-12T18:28:21.678 INFO:teuthology.orchestra.console:Power on smithi002 2023-03-12T18:28:25.936 INFO:teuthology.orchestra.console:Power on for smithi122 completed 2023-03-12T18:28:26.037 INFO:teuthology.provision.fog.smithi122:Waiting for deploy to finish 2023-03-12T18:28:30.070 INFO:teuthology.orchestra.console:Power on for smithi002 completed 2023-03-12T18:28:30.132 INFO:teuthology.orchestra.console:Power on for smithi178 completed 2023-03-12T18:28:30.171 INFO:teuthology.provision.fog.smithi002:Waiting for deploy to finish 2023-03-12T18:28:30.233 INFO:teuthology.provision.fog.smithi178:Waiting for deploy to finish 2023-03-12T18:32:32.581 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.122 2023-03-12T18:32:41.669 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.122 2023-03-12T18:32:50.725 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.122 2023-03-12T18:32:59.781 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.122 2023-03-12T18:33:08.837 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.122 2023-03-12T18:34:14.849 WARNING:teuthology.provision.fog:timed out 2023-03-12T18:34:21.150 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-03-12T18:34:21.461 INFO:teuthology.provision.fog.smithi122:Node is ready 2023-03-12T18:34:21.477 INFO:teuthology.orchestra.run.smithi122.stdout:smithi122.front.sepia.ceph.com 2023-03-12T18:34:21.534 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.15.122 smithi122.front.sepia.ceph.com smithi122 2023-03-12T18:34:22.006 INFO:teuthology.provision.fog.smithi122:Deploy complete! 2023-03-12T18:36:19.429 WARNING:teuthology.provision.fog:timed out 2023-03-12T18:36:35.717 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.102 2023-03-12T18:36:42.629 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.102 2023-03-12T18:36:51.685 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.102 2023-03-12T18:36:59.944 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-03-12T18:37:00.233 INFO:teuthology.provision.fog.smithi102:Node is ready 2023-03-12T18:37:00.249 INFO:teuthology.orchestra.run.smithi102.stdout:smithi022 2023-03-12T18:37:00.305 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.22 smithi022.front.sepia.ceph.com smithi022 2023-03-12T18:37:00.662 INFO:teuthology.provision.fog.smithi102:Deploy complete! 2023-03-12T18:38:40.709 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.178 2023-03-12T18:38:49.765 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.178 2023-03-12T18:38:58.821 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.178 2023-03-12T18:39:07.877 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.178 2023-03-12T18:39:16.933 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.178 2023-03-12T18:39:25.989 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.178 2023-03-12T18:39:35.045 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.178 2023-03-12T18:39:37.660 WARNING:teuthology.provision.fog:timed out 2023-03-12T18:39:44.101 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.178 2023-03-12T18:39:45.705 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.2 2023-03-12T18:39:54.757 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.2 2023-03-12T18:40:00.901 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.2 2023-03-12T18:40:06.982 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-03-12T18:40:07.053 INFO:teuthology.provision.fog.smithi002:Node is ready 2023-03-12T18:40:07.109 INFO:teuthology.orchestra.run.smithi002.stdout:smithi002 2023-03-12T18:40:07.168 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2 smithi002.front.sepia.ceph.com smithi002 2023-03-12T18:40:07.516 INFO:teuthology.provision.fog.smithi002:Deploy complete! 2023-03-12T18:40:50.149 WARNING:teuthology.provision.fog:timed out 2023-03-12T18:40:56.502 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-03-12T18:40:56.769 INFO:teuthology.provision.fog.smithi178:Node is ready 2023-03-12T18:40:56.787 INFO:teuthology.orchestra.run.smithi178.stdout:smithi178 2023-03-12T18:40:56.845 INFO:teuthology.orchestra.run.smithi178.stdout:172.21.15.178 smithi178.front.sepia.ceph.com smithi178 2023-03-12T18:40:57.188 INFO:teuthology.provision.fog.smithi178:Deploy complete! 2023-03-12T18:41:06.355 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 210, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/lock/ops.py", line 326, in reimage_machines reimaged = do_update_keys(list(reimaged.keys()))[1] File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/lock/ops.py", line 289, in do_update_keys keys_dict = misc.ssh_keyscan(machines, _raise=_raise) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1106, in ssh_keyscan while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/contextutil.py", line 135, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'ssh_keyscan smithi002.front.sepia.ceph.com' reached maximum tries (5) after waiting for 5 seconds 2023-03-12T18:41:06.445 INFO:teuthology.nuke:Checking targets against current locks 2023-03-12T18:41:06.499 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-12T18:41:06.528 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-12T18:41:06.561 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-12T18:41:06.604 INFO:teuthology.orchestra.console:Power off smithi002 2023-03-12T18:41:06.617 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-12T18:41:06.648 INFO:teuthology.orchestra.console:Power off smithi122 2023-03-12T18:41:06.669 INFO:teuthology.orchestra.console:Power off smithi102 2023-03-12T18:41:06.744 INFO:teuthology.orchestra.console:Power off smithi178 2023-03-12T18:41:06.782 INFO:teuthology.orchestra.console:Power off for smithi002 completed 2023-03-12T18:41:06.943 INFO:teuthology.lock.ops:unlocked: smithi002.front.sepia.ceph.com 2023-03-12T18:41:15.041 INFO:teuthology.orchestra.console:Power off for smithi122 completed 2023-03-12T18:41:15.117 INFO:teuthology.orchestra.console:Power off for smithi178 completed 2023-03-12T18:41:15.191 INFO:teuthology.lock.ops:unlocked: smithi122.front.sepia.ceph.com 2023-03-12T18:41:15.274 INFO:teuthology.lock.ops:unlocked: smithi178.front.sepia.ceph.com 2023-03-12T18:41:19.161 INFO:teuthology.orchestra.console:Power off for smithi102 completed 2023-03-12T18:41:19.317 INFO:teuthology.lock.ops:unlocked: smithi102.front.sepia.ceph.com 2023-03-12T18:41:19.767 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/__init__.py", line 64, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 52, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 210, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/lock/ops.py", line 326, in reimage_machines reimaged = do_update_keys(list(reimaged.keys()))[1] File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/lock/ops.py", line 289, in do_update_keys keys_dict = misc.ssh_keyscan(machines, _raise=_raise) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1106, in ssh_keyscan while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/contextutil.py", line 135, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'ssh_keyscan smithi002.front.sepia.ceph.com' reached maximum tries (5) after waiting for 5 seconds