2023-03-11T03:36:33.327 INFO:root:teuthology version: 0.0.1.dev42+ga6a4a7f 2023-03-11T03:36:33.428 INFO:teuthology.lock.ops:Start node 'smithi140.front.sepia.ceph.com' reimaging 2023-03-11T03:36:33.430 INFO:teuthology.lock.ops:Updating [smithi140.front.sepia.ceph.com]: reset os type and version on server 2023-03-11T03:36:33.430 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server 2023-03-11T03:36:33.450 INFO:teuthology.lock.ops:Node 'smithi140.front.sepia.ceph.com' reimaging is complete 2023-03-11T03:36:33.610 INFO:teuthology.provision.fog.smithi140:Scheduling deploy of rhel 8.6 2023-03-11T03:36:33.849 INFO:teuthology.orchestra.console:Power off smithi140 2023-03-11T03:36:33.996 INFO:teuthology.orchestra.console:Power off for smithi140 completed 2023-03-11T03:36:34.097 INFO:teuthology.orchestra.console:Power on smithi140 2023-03-11T03:36:42.462 INFO:teuthology.orchestra.console:Power on for smithi140 completed 2023-03-11T03:36:42.564 INFO:teuthology.provision.fog.smithi140:Waiting for deploy to finish 2023-03-11T03:39:28.948 ERROR:teuthology.orchestra.connection:Error authenticating with smithi140.front.sepia.ceph.com: Authentication failed. 2023-03-11T03:39:30.030 ERROR:teuthology.orchestra.connection:Error authenticating with smithi140.front.sepia.ceph.com: Authentication failed. 2023-03-11T03:39:31.147 ERROR:teuthology.orchestra.connection:Error authenticating with smithi140.front.sepia.ceph.com: Authentication failed. 2023-03-11T03:40:32.149 WARNING:teuthology.provision.fog:timed out 2023-03-11T03:41:38.153 WARNING:teuthology.provision.fog:timed out 2023-03-11T03:42:02.501 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:42:11.557 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:42:20.617 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:42:29.669 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:42:38.725 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:42:47.785 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:42:56.841 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:43:05.897 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:43:14.949 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:43:24.005 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:43:33.061 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:43:42.117 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:43:51.173 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:44:00.233 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:44:09.285 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:44:18.345 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:44:27.401 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:44:36.453 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:44:45.509 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:44:54.565 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:45:03.625 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:45:12.677 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:45:21.737 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:45:30.789 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:45:39.849 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:45:48.901 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:45:57.957 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:46:07.013 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:46:16.069 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:46:25.125 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:46:34.181 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:46:43.237 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:46:52.293 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:47:01.349 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:47:10.405 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:47:19.465 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:47:28.517 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:47:37.573 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:47:46.633 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:47:55.685 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:48:04.745 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:48:13.797 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:48:22.853 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:48:31.913 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:48:40.969 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:48:50.021 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:48:59.077 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:49:08.137 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:49:17.189 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:49:26.249 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:49:35.301 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:49:44.357 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:49:53.413 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:50:02.473 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:50:11.525 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:50:20.581 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:50:29.637 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:50:38.697 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:50:47.749 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:50:56.809 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:51:05.861 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:51:14.917 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:51:23.973 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:51:33.029 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:51:42.085 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:51:51.141 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:52:00.197 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:52:09.257 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:52:18.309 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:52:27.365 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:52:36.421 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:52:45.481 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:52:54.538 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:53:03.589 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:53:12.645 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:53:21.701 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:53:30.757 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:53:39.813 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:53:48.869 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:53:57.925 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:54:06.981 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:54:16.041 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:54:25.093 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:54:34.149 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:54:43.209 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:54:52.261 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:55:01.317 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:55:10.374 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:55:19.433 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:55:28.485 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:55:37.545 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:55:46.597 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:55:55.653 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:56:04.709 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:56:13.769 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:56:22.821 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:56:31.877 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:56:40.933 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-03-11T03:56:40.935 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/dispatcher/supervisor.py", line 209, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/lock/ops.py", line 324, in reimage_machines log.info("Node '%s' reimaging is complete", machine) File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/provision/__init__.py", line 39, in reimage return obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/provision/fog.py", line 271, in _wait_for_ready while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds 2023-03-11T03:56:41.005 INFO:teuthology.nuke:Checking targets against current locks 2023-03-11T03:56:41.062 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-11T03:56:41.139 INFO:teuthology.orchestra.console:Power off smithi140 2023-03-11T03:56:49.506 INFO:teuthology.orchestra.console:Power off for smithi140 completed 2023-03-11T03:56:49.662 INFO:teuthology.lock.ops:unlocked: smithi140.front.sepia.ceph.com 2023-03-11T03:56:49.785 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/dispatcher/__init__.py", line 63, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/dispatcher/supervisor.py", line 52, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/dispatcher/supervisor.py", line 209, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/lock/ops.py", line 324, in reimage_machines log.info("Node '%s' reimaging is complete", machine) File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/provision/__init__.py", line 39, in reimage return obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/provision/fog.py", line 271, in _wait_for_ready while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds