2023-01-24T14:04:35.355 INFO:root:teuthology version: 0.0.1.dev22+g8e1bba1 2023-01-24T14:04:35.490 INFO:teuthology.lock.ops:Start node 'smithi140.front.sepia.ceph.com' reimaging 2023-01-24T14:04:35.491 INFO:teuthology.lock.ops:Updating [smithi140.front.sepia.ceph.com]: reset os type and version on server 2023-01-24T14:04:35.491 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server 2023-01-24T14:04:35.509 INFO:teuthology.lock.ops:Node 'smithi140.front.sepia.ceph.com' reimaging is complete 2023-01-24T14:04:35.510 INFO:teuthology.lock.ops:Start node 'smithi145.front.sepia.ceph.com' reimaging 2023-01-24T14:04:35.510 INFO:teuthology.lock.ops:Updating [smithi145.front.sepia.ceph.com]: reset os type and version on server 2023-01-24T14:04:35.510 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server 2023-01-24T14:04:35.530 INFO:teuthology.lock.ops:Node 'smithi145.front.sepia.ceph.com' reimaging is complete 2023-01-24T14:04:35.676 INFO:teuthology.provision.fog.smithi140:Scheduling deploy of rhel 8.6 2023-01-24T14:04:35.681 INFO:teuthology.provision.fog.smithi145:Scheduling deploy of rhel 8.6 2023-01-24T14:04:35.931 INFO:teuthology.orchestra.console:Power off smithi140 2023-01-24T14:04:35.944 INFO:teuthology.orchestra.console:Power off smithi145 2023-01-24T14:04:36.072 INFO:teuthology.orchestra.console:Power off for smithi140 completed 2023-01-24T14:04:36.083 INFO:teuthology.orchestra.console:Power off for smithi145 completed 2023-01-24T14:04:36.174 INFO:teuthology.orchestra.console:Power on smithi140 2023-01-24T14:04:36.184 INFO:teuthology.orchestra.console:Power on smithi145 2023-01-24T14:04:44.544 INFO:teuthology.orchestra.console:Power on for smithi145 completed 2023-01-24T14:04:44.625 INFO:teuthology.orchestra.console:Power on for smithi140 completed 2023-01-24T14:04:44.646 INFO:teuthology.provision.fog.smithi145:Waiting for deploy to finish 2023-01-24T14:04:44.726 INFO:teuthology.provision.fog.smithi140:Waiting for deploy to finish 2023-01-24T14:07:33.992 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.145 2023-01-24T14:07:43.052 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.145 2023-01-24T14:07:49.096 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:07:52.136 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.145 2023-01-24T14:07:58.152 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:08:01.196 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.145 2023-01-24T14:08:07.208 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:08:10.284 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.145 2023-01-24T14:08:16.264 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:08:25.320 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:08:48.032 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-01-24T14:08:48.311 INFO:teuthology.provision.fog.smithi145:Node is ready 2023-01-24T14:08:48.336 INFO:teuthology.orchestra.run.smithi145.stdout:smithi022 2023-01-24T14:08:48.392 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.22 smithi022.front.sepia.ceph.com smithi022 2023-01-24T14:08:48.724 INFO:teuthology.provision.fog.smithi145:Deploy complete! 2023-01-24T14:09:31.368 WARNING:teuthology.provision.fog:timed out 2023-01-24T14:09:40.428 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:09:49.512 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:09:58.568 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:10:07.628 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:10:16.712 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:10:25.800 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:10:34.888 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:10:43.948 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:10:53.032 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:11:02.088 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:11:11.144 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:11:20.204 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:11:29.288 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:11:38.344 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:11:47.400 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:11:56.456 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:12:05.516 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:12:14.600 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:12:23.656 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:12:32.712 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:12:41.772 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:12:50.856 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:12:59.912 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:13:08.968 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:13:18.024 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:13:27.080 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:13:36.136 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:13:45.192 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:13:54.248 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:14:03.304 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:14:12.360 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:14:21.416 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:14:30.472 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:14:39.528 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:14:48.588 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:14:57.672 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:15:06.732 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:15:15.820 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:15:24.904 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:15:33.960 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:15:43.016 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:15:52.076 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:16:01.160 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:16:10.220 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:16:19.309 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:16:28.392 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:16:37.452 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:16:46.536 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:16:55.592 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:17:04.648 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:17:13.704 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:17:22.764 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:17:31.848 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:17:40.904 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:17:49.964 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:17:59.048 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:18:08.104 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:18:17.160 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:18:26.220 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:18:35.308 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:18:44.396 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:18:53.480 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:19:02.572 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:19:11.660 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:19:20.744 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:19:29.800 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:19:38.856 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:19:47.912 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:19:56.968 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:20:06.024 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:20:15.088 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:20:24.168 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:20:33.224 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:20:42.312 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:20:51.372 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:21:00.456 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:21:09.516 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:21:18.600 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:21:27.660 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:21:36.748 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:21:45.832 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:21:54.892 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:22:03.976 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:22:13.036 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:22:22.120 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:22:31.176 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:22:40.235 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:22:49.320 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:22:58.379 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:23:07.464 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:23:16.527 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:23:25.608 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:23:34.664 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:23:43.752 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-24T14:23:43.754 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/dispatcher/supervisor.py", line 209, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/lock/ops.py", line 324, in reimage_machines log.info("Node '%s' reimaging is complete", machine) File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/provision/__init__.py", line 39, in reimage return obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/provision/fog.py", line 271, in _wait_for_ready while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds 2023-01-24T14:23:43.828 INFO:teuthology.nuke:Checking targets against current locks 2023-01-24T14:23:43.882 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-24T14:23:43.908 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-24T14:23:43.961 INFO:teuthology.orchestra.console:Power off smithi140 2023-01-24T14:23:43.993 INFO:teuthology.orchestra.console:Power off smithi145 2023-01-24T14:23:52.355 INFO:teuthology.orchestra.console:Power off for smithi145 completed 2023-01-24T14:23:52.496 INFO:teuthology.lock.ops:unlocked: smithi145.front.sepia.ceph.com 2023-01-24T14:23:56.436 INFO:teuthology.orchestra.console:Power off for smithi140 completed 2023-01-24T14:23:56.589 INFO:teuthology.lock.ops:unlocked: smithi140.front.sepia.ceph.com 2023-01-24T14:23:56.700 ERROR:teuthology.dispatcher.supervisor:Reimage failed 10 times ... marking machine down 2023-01-24T14:23:56.796 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/dispatcher/__init__.py", line 63, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/dispatcher/supervisor.py", line 52, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/dispatcher/supervisor.py", line 209, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/lock/ops.py", line 324, in reimage_machines log.info("Node '%s' reimaging is complete", machine) File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/provision/__init__.py", line 39, in reimage return obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/provision/fog.py", line 271, in _wait_for_ready while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds