2023-03-08T20:59:24.442 INFO:root:teuthology version: 0.0.1.dev51+ga2dfb05 2023-03-08T20:59:24.594 INFO:teuthology.lock.ops:Start node 'smithi038.front.sepia.ceph.com' reimaging 2023-03-08T20:59:24.596 INFO:teuthology.lock.ops:Updating [smithi038.front.sepia.ceph.com]: reset os type and version on server 2023-03-08T20:59:24.596 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2023-03-08T20:59:24.624 INFO:teuthology.lock.ops:Node 'smithi038.front.sepia.ceph.com' reimaging is complete 2023-03-08T20:59:24.625 INFO:teuthology.lock.ops:Start node 'smithi140.front.sepia.ceph.com' reimaging 2023-03-08T20:59:24.625 INFO:teuthology.lock.ops:Updating [smithi140.front.sepia.ceph.com]: reset os type and version on server 2023-03-08T20:59:24.626 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server 2023-03-08T20:59:24.655 INFO:teuthology.lock.ops:Node 'smithi140.front.sepia.ceph.com' reimaging is complete 2023-03-08T20:59:24.790 INFO:teuthology.provision.fog.smithi038:Scheduling deploy of rhel 8.6 2023-03-08T20:59:24.827 INFO:teuthology.provision.fog.smithi140:Scheduling deploy of rhel 8.6 2023-03-08T20:59:25.046 INFO:teuthology.orchestra.console:Power off smithi038 2023-03-08T20:59:25.069 INFO:teuthology.orchestra.console:Power off smithi140 2023-03-08T20:59:33.441 INFO:teuthology.orchestra.console:Power off for smithi038 completed 2023-03-08T20:59:33.543 INFO:teuthology.orchestra.console:Power on smithi038 2023-03-08T20:59:37.565 INFO:teuthology.orchestra.console:Power off for smithi140 completed 2023-03-08T20:59:37.667 INFO:teuthology.orchestra.console:Power on smithi140 2023-03-08T20:59:41.915 INFO:teuthology.orchestra.console:Power on for smithi038 completed 2023-03-08T20:59:42.017 INFO:teuthology.provision.fog.smithi038:Waiting for deploy to finish 2023-03-08T20:59:46.040 INFO:teuthology.orchestra.console:Power on for smithi140 completed 2023-03-08T20:59:46.141 INFO:teuthology.provision.fog.smithi140:Waiting for deploy to finish 2023-03-08T21:02:15.877 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.38 2023-03-08T21:02:24.965 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.38 2023-03-08T21:02:34.021 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.38 2023-03-08T21:02:34.981 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:02:43.081 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.38 2023-03-08T21:02:44.037 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:02:52.169 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.38 2023-03-08T21:02:53.097 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:03:01.257 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.38 2023-03-08T21:03:02.181 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:03:11.237 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:03:20.297 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:03:29.385 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:03:38.469 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:04:07.285 WARNING:teuthology.provision.fog:timed out 2023-03-08T21:04:13.630 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-03-08T21:04:13.912 INFO:teuthology.provision.fog.smithi038:Node is ready 2023-03-08T21:04:13.976 INFO:teuthology.orchestra.run.smithi038.stdout:smithi038.front.sepia.ceph.com 2023-03-08T21:04:14.037 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38 smithi038.front.sepia.ceph.com smithi038 2023-03-08T21:04:14.377 INFO:teuthology.provision.fog.smithi038:Deploy complete! 2023-03-08T21:04:44.502 WARNING:teuthology.provision.fog:timed out 2023-03-08T21:05:08.869 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:05:17.961 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:05:27.045 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:05:36.101 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:05:45.157 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:05:54.213 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:06:03.269 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:06:12.325 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:06:21.381 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:06:30.437 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:06:39.493 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:06:48.553 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:06:57.637 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:07:06.693 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:07:15.749 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:07:24.805 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:07:33.861 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:07:42.949 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:07:52.005 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:08:01.093 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:08:10.149 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:08:19.205 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:08:28.265 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:08:37.353 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:08:46.437 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:08:55.493 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:09:04.553 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:09:13.637 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:09:22.693 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:09:31.749 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:09:40.805 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:09:49.861 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:09:58.917 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:10:07.973 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:10:17.029 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:10:26.085 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:10:35.141 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:10:44.197 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:10:53.253 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:11:02.341 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:11:11.429 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:11:20.517 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:11:29.577 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:11:38.661 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:11:47.721 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:11:56.809 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:12:05.893 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:12:14.949 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:12:24.005 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:12:33.061 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:12:42.117 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:12:51.177 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:13:00.265 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:13:09.349 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:13:18.405 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:13:27.461 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:13:36.549 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:13:45.609 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:13:54.697 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:14:03.785 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:14:12.869 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:14:21.957 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:14:31.014 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:14:40.069 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:14:49.125 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:14:58.185 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:15:07.269 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:15:16.325 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:15:25.413 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:15:34.477 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:15:43.561 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:15:52.649 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:16:01.733 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:16:10.794 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:16:19.877 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:16:28.933 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:16:37.989 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:16:47.048 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:16:56.133 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:17:05.193 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:17:14.281 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:17:23.365 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:17:32.425 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:17:41.513 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:17:50.597 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:17:59.653 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:18:08.709 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:18:17.765 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:18:26.857 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:18:35.941 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:18:44.997 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-08T21:18:44.999 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/dispatcher/supervisor.py", line 209, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/lock/ops.py", line 324, in reimage_machines log.info("Node '%s' reimaging is complete", machine) File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/provision/__init__.py", line 39, in reimage return obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/provision/fog.py", line 271, in _wait_for_ready while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/contextutil.py", line 135, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds 2023-03-08T21:18:45.088 INFO:teuthology.nuke:Checking targets against current locks 2023-03-08T21:18:45.135 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-08T21:18:45.168 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-08T21:18:45.223 INFO:teuthology.orchestra.console:Power off smithi038 2023-03-08T21:18:45.266 INFO:teuthology.orchestra.console:Power off smithi140 2023-03-08T21:18:53.606 INFO:teuthology.orchestra.console:Power off for smithi038 completed 2023-03-08T21:18:53.766 INFO:teuthology.lock.ops:unlocked: smithi038.front.sepia.ceph.com 2023-03-08T21:18:57.752 INFO:teuthology.orchestra.console:Power off for smithi140 completed 2023-03-08T21:18:57.907 INFO:teuthology.lock.ops:unlocked: smithi140.front.sepia.ceph.com 2023-03-08T21:18:58.122 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/dispatcher/__init__.py", line 63, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/dispatcher/supervisor.py", line 52, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/dispatcher/supervisor.py", line 209, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/lock/ops.py", line 324, in reimage_machines log.info("Node '%s' reimaging is complete", machine) File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/provision/__init__.py", line 39, in reimage return obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/provision/fog.py", line 271, in _wait_for_ready while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/contextutil.py", line 135, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds