2022-12-21T09:43:53.890 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2022-12-21T09:43:54.032 INFO:teuthology.lock.ops:Start node 'smithi140.front.sepia.ceph.com' reimaging 2022-12-21T09:43:54.033 INFO:teuthology.lock.ops:Updating [smithi140.front.sepia.ceph.com]: reset os type and version on server 2022-12-21T09:43:54.033 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server 2022-12-21T09:43:54.055 INFO:teuthology.lock.ops:Node 'smithi140.front.sepia.ceph.com' reimaging is complete 2022-12-21T09:43:54.056 INFO:teuthology.lock.ops:Start node 'smithi151.front.sepia.ceph.com' reimaging 2022-12-21T09:43:54.056 INFO:teuthology.lock.ops:Updating [smithi151.front.sepia.ceph.com]: reset os type and version on server 2022-12-21T09:43:54.056 INFO:teuthology.lock.ops:Updating smithi151.front.sepia.ceph.com on lock server 2022-12-21T09:43:54.076 INFO:teuthology.lock.ops:Node 'smithi151.front.sepia.ceph.com' reimaging is complete 2022-12-21T09:43:54.221 INFO:teuthology.provision.fog.smithi140:Scheduling deploy of rhel 8.6 2022-12-21T09:43:54.243 INFO:teuthology.provision.fog.smithi151:Scheduling deploy of rhel 8.6 2022-12-21T09:43:54.488 INFO:teuthology.orchestra.console:Power off smithi140 2022-12-21T09:43:54.505 INFO:teuthology.orchestra.console:Power off smithi151 2022-12-21T09:43:54.637 INFO:teuthology.orchestra.console:Power off for smithi140 completed 2022-12-21T09:43:54.737 INFO:teuthology.orchestra.console:Power on smithi140 2022-12-21T09:44:03.102 INFO:teuthology.orchestra.console:Power on for smithi140 completed 2022-12-21T09:44:03.202 INFO:teuthology.provision.fog.smithi140:Waiting for deploy to finish 2022-12-21T09:44:06.998 INFO:teuthology.orchestra.console:Power off for smithi151 completed 2022-12-21T09:44:07.099 INFO:teuthology.orchestra.console:Power on smithi151 2022-12-21T09:44:15.470 INFO:teuthology.orchestra.console:Power on for smithi151 completed 2022-12-21T09:44:15.571 INFO:teuthology.provision.fog.smithi151:Waiting for deploy to finish 2022-12-21T09:47:07.275 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:47:16.359 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:47:19.625 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.151 2022-12-21T09:47:25.415 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:47:28.679 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.151 2022-12-21T09:47:34.471 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:47:37.735 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.151 2022-12-21T09:47:43.527 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:47:46.795 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.151 2022-12-21T09:47:52.583 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:47:55.851 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.151 2022-12-21T09:48:01.643 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:48:04.903 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.151 2022-12-21T09:48:10.728 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:48:35.207 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:48:44.263 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:48:53.319 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:49:02.375 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:49:10.910 WARNING:teuthology.provision.fog:timed out 2022-12-21T09:49:11.431 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:49:17.258 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-12-21T09:49:17.550 INFO:teuthology.provision.fog.smithi151:Node is ready 2022-12-21T09:49:17.568 INFO:teuthology.orchestra.run.smithi151.stdout:smithi151.front.sepia.ceph.com 2022-12-21T09:49:17.632 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151 smithi151.front.sepia.ceph.com smithi151 2022-12-21T09:49:18.061 INFO:teuthology.provision.fog.smithi151:Deploy complete! 2022-12-21T09:49:20.487 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:49:29.543 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:49:38.631 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:49:47.687 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:49:56.743 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:50:05.803 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:50:14.888 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:50:23.943 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:50:33.003 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:50:42.087 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:50:51.147 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:51:00.231 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:51:09.296 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:51:18.375 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:51:27.435 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:51:36.523 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:51:45.607 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:51:54.663 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:52:03.751 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:52:12.807 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:52:21.867 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:52:30.951 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:52:40.007 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:52:49.063 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:52:58.119 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:53:07.175 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:53:16.235 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:53:25.319 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:53:34.375 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:53:43.435 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:53:52.519 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:54:01.575 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:54:10.635 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:54:19.723 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:54:28.807 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:54:37.867 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:54:46.955 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:54:56.040 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:55:05.127 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:55:14.187 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:55:23.271 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:55:32.327 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:55:41.387 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:55:50.471 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:55:59.527 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:56:08.839 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:56:17.899 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:56:26.983 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:56:36.039 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:56:45.095 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:56:54.151 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:57:03.691 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:57:12.775 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:57:21.835 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:57:30.919 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:57:39.975 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:57:49.035 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:57:58.119 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:58:07.175 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:58:16.263 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:58:25.319 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:58:34.379 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:58:43.467 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:58:52.555 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:59:01.639 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:59:10.699 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:59:19.783 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:59:28.839 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:59:37.895 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:59:46.955 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T09:59:56.039 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T10:00:05.099 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T10:00:14.184 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T10:00:23.239 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T10:00:32.295 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T10:00:41.351 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T10:00:50.407 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T10:00:59.463 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T10:01:08.523 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T10:01:17.607 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T10:01:26.667 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T10:01:35.751 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T10:01:44.807 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T10:01:53.863 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T10:02:02.923 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T10:02:12.007 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T10:02:21.063 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-21T10:02:21.067 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 209, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/lock/ops.py", line 324, in reimage_machines log.info("Node '%s' reimaging is complete", machine) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/provision/__init__.py", line 39, in reimage return obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/provision/fog.py", line 271, in _wait_for_ready while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds 2022-12-21T10:02:21.150 INFO:teuthology.nuke:Checking targets against current locks 2022-12-21T10:02:21.200 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-12-21T10:02:21.245 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-12-21T10:02:21.331 INFO:teuthology.orchestra.console:Power off smithi140 2022-12-21T10:02:21.354 INFO:teuthology.orchestra.console:Power off smithi151 2022-12-21T10:02:29.743 INFO:teuthology.orchestra.console:Power off for smithi140 completed 2022-12-21T10:02:29.753 INFO:teuthology.orchestra.console:Power off for smithi151 completed 2022-12-21T10:02:29.928 INFO:teuthology.lock.ops:unlocked: smithi140.front.sepia.ceph.com 2022-12-21T10:02:29.929 INFO:teuthology.lock.ops:unlocked: smithi151.front.sepia.ceph.com 2022-12-21T10:02:36.302 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/__init__.py", line 63, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 52, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 209, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/lock/ops.py", line 324, in reimage_machines log.info("Node '%s' reimaging is complete", machine) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/provision/__init__.py", line 39, in reimage return obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/provision/fog.py", line 271, in _wait_for_ready while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds