2023-05-05T14:24:15.593 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-05T14:24:15.828 INFO:teuthology.lock.ops:Start node 'smithi008.front.sepia.ceph.com' reimaging 2023-05-05T14:24:15.829 INFO:teuthology.lock.ops:Updating [smithi008.front.sepia.ceph.com]: reset os type and version on server 2023-05-05T14:24:15.830 INFO:teuthology.lock.ops:Updating smithi008.front.sepia.ceph.com on lock server 2023-05-05T14:24:15.852 INFO:teuthology.lock.ops:Node 'smithi008.front.sepia.ceph.com' reimaging is complete 2023-05-05T14:24:15.853 INFO:teuthology.lock.ops:Start node 'smithi047.front.sepia.ceph.com' reimaging 2023-05-05T14:24:15.853 INFO:teuthology.lock.ops:Updating [smithi047.front.sepia.ceph.com]: reset os type and version on server 2023-05-05T14:24:15.853 INFO:teuthology.lock.ops:Updating smithi047.front.sepia.ceph.com on lock server 2023-05-05T14:24:15.877 INFO:teuthology.lock.ops:Node 'smithi047.front.sepia.ceph.com' reimaging is complete 2023-05-05T14:24:15.877 INFO:teuthology.lock.ops:Start node 'smithi071.front.sepia.ceph.com' reimaging 2023-05-05T14:24:15.878 INFO:teuthology.lock.ops:Updating [smithi071.front.sepia.ceph.com]: reset os type and version on server 2023-05-05T14:24:15.878 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2023-05-05T14:24:15.902 INFO:teuthology.lock.ops:Node 'smithi071.front.sepia.ceph.com' reimaging is complete 2023-05-05T14:24:15.902 INFO:teuthology.lock.ops:Start node 'smithi092.front.sepia.ceph.com' reimaging 2023-05-05T14:24:15.903 INFO:teuthology.lock.ops:Updating [smithi092.front.sepia.ceph.com]: reset os type and version on server 2023-05-05T14:24:15.903 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server 2023-05-05T14:24:16.027 INFO:teuthology.provision.fog.smithi008:Scheduling deploy of ubuntu 20.04 2023-05-05T14:24:16.047 INFO:teuthology.provision.fog.smithi047:Scheduling deploy of ubuntu 20.04 2023-05-05T14:24:16.069 INFO:teuthology.provision.fog.smithi071:Scheduling deploy of ubuntu 20.04 2023-05-05T14:24:16.351 INFO:teuthology.orchestra.console:Power off smithi047 2023-05-05T14:24:16.365 INFO:teuthology.orchestra.console:Power off smithi008 2023-05-05T14:24:16.404 INFO:teuthology.orchestra.console:Power off smithi071 2023-05-05T14:24:16.513 INFO:teuthology.orchestra.console:Power off for smithi047 completed 2023-05-05T14:24:16.514 INFO:teuthology.orchestra.console:Power off for smithi008 completed 2023-05-05T14:24:16.548 INFO:teuthology.orchestra.console:Power off for smithi071 completed 2023-05-05T14:24:16.614 INFO:teuthology.orchestra.console:Power on smithi047 2023-05-05T14:24:16.623 INFO:teuthology.orchestra.console:Power on smithi008 2023-05-05T14:24:16.653 INFO:teuthology.orchestra.console:Power on smithi071 2023-05-05T14:24:16.944 INFO:teuthology.lock.ops:Node 'smithi092.front.sepia.ceph.com' reimaging is complete 2023-05-05T14:24:16.945 INFO:teuthology.lock.ops:Start node 'smithi125.front.sepia.ceph.com' reimaging 2023-05-05T14:24:16.945 INFO:teuthology.lock.ops:Updating [smithi125.front.sepia.ceph.com]: reset os type and version on server 2023-05-05T14:24:16.945 INFO:teuthology.lock.ops:Updating smithi125.front.sepia.ceph.com on lock server 2023-05-05T14:24:16.960 INFO:teuthology.lock.ops:Node 'smithi125.front.sepia.ceph.com' reimaging is complete 2023-05-05T14:24:17.105 INFO:teuthology.provision.fog.smithi092:Scheduling deploy of ubuntu 20.04 2023-05-05T14:24:17.120 INFO:teuthology.provision.fog.smithi125:Scheduling deploy of ubuntu 20.04 2023-05-05T14:24:17.484 INFO:teuthology.orchestra.console:Power off smithi092 2023-05-05T14:24:17.500 INFO:teuthology.orchestra.console:Power off smithi125 2023-05-05T14:24:17.635 INFO:teuthology.orchestra.console:Power off for smithi092 completed 2023-05-05T14:24:17.737 INFO:teuthology.orchestra.console:Power on smithi092 2023-05-05T14:24:20.884 INFO:teuthology.orchestra.console:Power on for smithi008 completed 2023-05-05T14:24:20.907 INFO:teuthology.orchestra.console:Power on for smithi071 completed 2023-05-05T14:24:20.990 INFO:teuthology.provision.fog.smithi008:Waiting for deploy to finish 2023-05-05T14:24:21.008 INFO:teuthology.provision.fog.smithi071:Waiting for deploy to finish 2023-05-05T14:24:24.988 INFO:teuthology.orchestra.console:Power on for smithi047 completed 2023-05-05T14:24:25.089 INFO:teuthology.provision.fog.smithi047:Waiting for deploy to finish 2023-05-05T14:24:26.100 INFO:teuthology.orchestra.console:Power on for smithi092 completed 2023-05-05T14:24:26.201 INFO:teuthology.provision.fog.smithi092:Waiting for deploy to finish 2023-05-05T14:24:29.981 INFO:teuthology.orchestra.console:Power off for smithi125 completed 2023-05-05T14:24:30.082 INFO:teuthology.orchestra.console:Power on smithi125 2023-05-05T14:24:38.447 INFO:teuthology.orchestra.console:Power on for smithi125 completed 2023-05-05T14:24:38.548 INFO:teuthology.provision.fog.smithi125:Waiting for deploy to finish 2023-05-05T14:28:01.317 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.47 2023-05-05T14:28:10.405 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.47 2023-05-05T14:28:19.461 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.47 2023-05-05T14:28:28.517 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.47 2023-05-05T14:28:37.573 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.47 2023-05-05T14:29:43.577 WARNING:teuthology.provision.fog:timed out 2023-05-05T14:29:49.779 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-05-05T14:29:50.283 INFO:teuthology.provision.fog.smithi047:Node is ready 2023-05-05T14:29:50.289 INFO:teuthology.orchestra.run.smithi047.stdout:smithi047.front.sepia.ceph.com 2023-05-05T14:29:50.296 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.47 smithi047.front.sepia.ceph.com smithi047 2023-05-05T14:29:50.506 INFO:teuthology.provision.fog.smithi047:Deploy complete! 2023-05-05T14:34:39.141 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.92 2023-05-05T14:34:48.197 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.92 2023-05-05T14:34:57.285 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.92 2023-05-05T14:35:06.341 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.92 2023-05-05T14:35:15.397 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.92 2023-05-05T14:35:24.453 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.92 2023-05-05T14:35:33.509 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.92 2023-05-05T14:35:58.085 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.92 2023-05-05T14:36:07.141 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.92 2023-05-05T14:36:14.415 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-05-05T14:36:35.403 INFO:teuthology.provision.fog.smithi092:Node is ready 2023-05-05T14:36:35.409 INFO:teuthology.orchestra.run.smithi092.stdout:smithi092.front.sepia.ceph.com 2023-05-05T14:36:35.416 INFO:teuthology.orchestra.run.smithi092.stdout:172.21.15.92 smithi092.front.sepia.ceph.com smithi092 2023-05-05T14:36:35.590 INFO:teuthology.provision.fog.smithi092:Deploy complete! 2023-05-05T14:36:50.565 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.8 2023-05-05T14:36:59.621 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.8 2023-05-05T14:37:08.677 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.8 2023-05-05T14:37:17.733 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.8 2023-05-05T14:37:26.789 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.8 2023-05-05T14:38:07.973 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.125 2023-05-05T14:38:17.029 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.125 2023-05-05T14:38:26.085 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.125 2023-05-05T14:38:32.793 WARNING:teuthology.provision.fog:timed out 2023-05-05T14:38:35.173 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.125 2023-05-05T14:38:39.006 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-05-05T14:38:39.541 INFO:teuthology.provision.fog.smithi008:Node is ready 2023-05-05T14:38:39.547 INFO:teuthology.orchestra.run.smithi008.stdout:smithi008.front.sepia.ceph.com 2023-05-05T14:38:39.595 INFO:teuthology.orchestra.run.smithi008.stdout:172.21.15.8 smithi008.front.sepia.ceph.com smithi008 2023-05-05T14:38:39.778 INFO:teuthology.provision.fog.smithi008:Deploy complete! 2023-05-05T14:38:44.229 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.125 2023-05-05T14:38:53.285 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.125 2023-05-05T14:39:18.581 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/dispatcher/supervisor.py", line 226, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/lock/ops.py", line 324, in reimage_machines log.info("Node '%s' reimaging is complete", machine) File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/provision/__init__.py", line 39, in reimage return obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/provision/fog.py", line 86, in create self.wait_for_deploy_task(task_id) File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/provision/fog.py", line 255, in wait_for_deploy_task while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/contextutil.py", line 135, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (60) after waiting for 900 seconds 2023-05-05T14:39:18.687 INFO:teuthology.nuke:Checking targets against current locks 2023-05-05T14:39:18.739 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-05-05T14:39:18.771 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-05-05T14:39:18.792 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-05-05T14:39:18.811 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-05-05T14:39:18.825 INFO:teuthology.orchestra.console:Power off smithi008 2023-05-05T14:39:18.839 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-05-05T14:39:18.878 INFO:teuthology.orchestra.console:Power off smithi047 2023-05-05T14:39:18.925 INFO:teuthology.orchestra.console:Power off smithi092 2023-05-05T14:39:18.942 INFO:teuthology.orchestra.console:Power off smithi071 2023-05-05T14:39:18.983 INFO:teuthology.orchestra.console:Power off smithi125 2023-05-05T14:39:27.228 INFO:teuthology.orchestra.console:Power off for smithi008 completed 2023-05-05T14:39:27.282 INFO:teuthology.orchestra.console:Power off for smithi047 completed 2023-05-05T14:39:27.317 INFO:teuthology.orchestra.console:Power off for smithi092 completed 2023-05-05T14:39:27.358 INFO:teuthology.orchestra.console:Power off for smithi125 completed 2023-05-05T14:39:27.380 INFO:teuthology.lock.ops:unlocked: smithi008.front.sepia.ceph.com 2023-05-05T14:39:27.422 INFO:teuthology.lock.ops:unlocked: smithi047.front.sepia.ceph.com 2023-05-05T14:39:27.471 INFO:teuthology.lock.ops:unlocked: smithi092.front.sepia.ceph.com 2023-05-05T14:39:27.514 INFO:teuthology.lock.ops:unlocked: smithi125.front.sepia.ceph.com 2023-05-05T14:39:31.436 INFO:teuthology.orchestra.console:Power off for smithi071 completed 2023-05-05T14:39:31.589 INFO:teuthology.lock.ops:unlocked: smithi071.front.sepia.ceph.com 2023-05-05T14:39:32.690 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/dispatcher/supervisor.py", line 56, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/dispatcher/supervisor.py", line 226, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/lock/ops.py", line 324, in reimage_machines log.info("Node '%s' reimaging is complete", machine) File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/provision/__init__.py", line 39, in reimage return obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/provision/fog.py", line 86, in create self.wait_for_deploy_task(task_id) File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/provision/fog.py", line 255, in wait_for_deploy_task while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/contextutil.py", line 135, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (60) after waiting for 900 seconds