2023-01-13T11:22:02.639 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2023-01-13T11:22:02.777 INFO:teuthology.lock.ops:Start node 'smithi140.front.sepia.ceph.com' reimaging 2023-01-13T11:22:02.779 INFO:teuthology.lock.ops:Updating [smithi140.front.sepia.ceph.com]: reset os type and version on server 2023-01-13T11:22:02.779 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server 2023-01-13T11:22:02.800 INFO:teuthology.lock.ops:Node 'smithi140.front.sepia.ceph.com' reimaging is complete 2023-01-13T11:22:02.800 INFO:teuthology.lock.ops:Start node 'smithi187.front.sepia.ceph.com' reimaging 2023-01-13T11:22:02.801 INFO:teuthology.lock.ops:Updating [smithi187.front.sepia.ceph.com]: reset os type and version on server 2023-01-13T11:22:02.801 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2023-01-13T11:22:02.818 INFO:teuthology.lock.ops:Node 'smithi187.front.sepia.ceph.com' reimaging is complete 2023-01-13T11:22:02.960 INFO:teuthology.provision.fog.smithi140:Scheduling deploy of rhel 8.6 2023-01-13T11:22:02.995 INFO:teuthology.provision.fog.smithi187:Scheduling deploy of rhel 8.6 2023-01-13T11:22:03.217 INFO:teuthology.orchestra.console:Power off smithi140 2023-01-13T11:22:03.260 INFO:teuthology.orchestra.console:Power off smithi187 2023-01-13T11:22:03.370 INFO:teuthology.orchestra.console:Power off for smithi140 completed 2023-01-13T11:22:03.411 INFO:teuthology.orchestra.console:Power off for smithi187 completed 2023-01-13T11:22:03.472 INFO:teuthology.orchestra.console:Power on smithi140 2023-01-13T11:22:03.512 INFO:teuthology.orchestra.console:Power on smithi187 2023-01-13T11:22:07.736 INFO:teuthology.orchestra.console:Power on for smithi140 completed 2023-01-13T11:22:07.838 INFO:teuthology.provision.fog.smithi140:Waiting for deploy to finish 2023-01-13T11:22:11.887 INFO:teuthology.orchestra.console:Power on for smithi187 completed 2023-01-13T11:22:11.988 INFO:teuthology.provision.fog.smithi187:Waiting for deploy to finish 2023-01-13T11:24:56.755 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:25:00.844 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-01-13T11:25:05.840 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:25:09.904 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-01-13T11:25:14.928 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:25:18.988 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-01-13T11:25:24.012 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:25:28.048 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-01-13T11:25:33.068 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:25:37.132 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-01-13T11:25:42.124 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:25:46.188 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-01-13T11:25:51.180 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:26:24.059 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-01-13T11:26:24.489 INFO:teuthology.provision.fog.smithi187:Node is ready 2023-01-13T11:26:24.506 INFO:teuthology.orchestra.run.smithi187.stdout:smithi187.front.sepia.ceph.com 2023-01-13T11:26:24.563 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.15.187 smithi187.front.sepia.ceph.com smithi187 2023-01-13T11:26:25.000 INFO:teuthology.provision.fog.smithi187:Deploy complete! 2023-01-13T11:26:57.216 WARNING:teuthology.provision.fog:timed out 2023-01-13T11:27:06.284 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:27:15.344 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:27:24.428 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:27:33.488 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:27:42.572 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:27:51.628 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:28:00.684 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:28:09.740 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:28:18.796 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:28:27.856 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:28:36.940 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:28:45.996 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:28:55.052 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:29:04.112 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:29:13.196 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:29:22.288 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:29:31.372 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:29:40.428 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:29:49.488 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:29:58.572 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:30:07.628 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:30:16.688 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:30:25.772 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:30:34.832 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:30:43.916 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:30:53.008 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:31:02.092 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:31:11.148 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:31:20.204 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:31:29.260 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:31:38.316 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:31:47.372 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:31:56.432 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:32:05.516 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:32:14.572 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:32:23.628 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:32:32.716 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:32:41.808 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:32:50.892 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:32:59.952 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:33:09.036 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:33:18.092 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:33:27.148 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:33:36.204 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:33:45.264 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:33:54.348 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:34:03.404 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:34:12.464 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:34:21.552 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:34:30.636 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:34:39.696 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:34:48.784 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:34:57.872 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:35:06.956 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:35:16.012 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:35:25.104 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:35:34.192 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:35:43.280 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:35:52.364 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:36:01.420 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:36:10.476 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:36:19.532 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:36:28.588 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:36:37.644 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:36:46.732 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:36:55.788 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:37:04.844 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:37:13.900 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:37:22.956 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:37:32.012 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:37:41.072 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:37:50.156 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:37:59.212 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:38:08.272 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:38:17.356 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:38:26.412 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:38:35.472 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:38:44.560 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:38:53.644 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:39:02.705 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:39:11.792 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:39:20.876 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:39:29.964 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:39:39.020 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:39:48.080 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:39:57.164 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:40:06.252 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:40:15.312 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:40:24.396 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:40:33.456 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:40:42.540 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:40:51.597 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-13T11:40:51.600 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 209, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/lock/ops.py", line 324, in reimage_machines log.info("Node '%s' reimaging is complete", machine) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/provision/__init__.py", line 39, in reimage return obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/provision/fog.py", line 271, in _wait_for_ready while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds 2023-01-13T11:40:51.671 INFO:teuthology.nuke:Checking targets against current locks 2023-01-13T11:40:51.721 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-13T11:40:51.746 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-13T11:40:51.807 INFO:teuthology.orchestra.console:Power off smithi140 2023-01-13T11:40:51.846 INFO:teuthology.orchestra.console:Power off smithi187 2023-01-13T11:41:00.216 INFO:teuthology.orchestra.console:Power off for smithi187 completed 2023-01-13T11:41:00.361 INFO:teuthology.lock.ops:unlocked: smithi187.front.sepia.ceph.com 2023-01-13T11:41:04.296 INFO:teuthology.orchestra.console:Power off for smithi140 completed 2023-01-13T11:41:04.446 INFO:teuthology.lock.ops:unlocked: smithi140.front.sepia.ceph.com 2023-01-13T11:41:07.745 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/__init__.py", line 63, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 52, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 209, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/lock/ops.py", line 324, in reimage_machines log.info("Node '%s' reimaging is complete", machine) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/provision/__init__.py", line 39, in reimage return obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/provision/fog.py", line 271, in _wait_for_ready while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds