2022-12-14T07:55:10.495 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2022-12-14T07:55:10.669 INFO:teuthology.lock.ops:Start node 'smithi111.front.sepia.ceph.com' reimaging 2022-12-14T07:55:10.670 INFO:teuthology.lock.ops:Updating [smithi111.front.sepia.ceph.com]: reset os type and version on server 2022-12-14T07:55:10.670 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server 2022-12-14T07:55:10.703 INFO:teuthology.lock.ops:Node 'smithi111.front.sepia.ceph.com' reimaging is complete 2022-12-14T07:55:10.704 INFO:teuthology.lock.ops:Start node 'smithi140.front.sepia.ceph.com' reimaging 2022-12-14T07:55:10.705 INFO:teuthology.lock.ops:Updating [smithi140.front.sepia.ceph.com]: reset os type and version on server 2022-12-14T07:55:10.705 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server 2022-12-14T07:55:10.729 INFO:teuthology.lock.ops:Node 'smithi140.front.sepia.ceph.com' reimaging is complete 2022-12-14T07:55:10.943 INFO:teuthology.provision.fog.smithi140:Scheduling deploy of rhel 8.6 2022-12-14T07:55:10.949 INFO:teuthology.provision.fog.smithi111:Scheduling deploy of rhel 8.6 2022-12-14T07:55:11.334 INFO:teuthology.orchestra.console:Power off smithi140 2022-12-14T07:55:11.356 INFO:teuthology.orchestra.console:Power off smithi111 2022-12-14T07:55:23.864 INFO:teuthology.orchestra.console:Power off for smithi140 completed 2022-12-14T07:55:23.889 INFO:teuthology.orchestra.console:Power off for smithi111 completed 2022-12-14T07:55:23.966 INFO:teuthology.orchestra.console:Power on smithi140 2022-12-14T07:55:23.990 INFO:teuthology.orchestra.console:Power on smithi111 2022-12-14T07:55:28.275 INFO:teuthology.orchestra.console:Power on for smithi140 completed 2022-12-14T07:55:28.278 INFO:teuthology.orchestra.console:Power on for smithi111 completed 2022-12-14T07:55:28.377 INFO:teuthology.provision.fog.smithi140:Waiting for deploy to finish 2022-12-14T07:55:28.381 INFO:teuthology.provision.fog.smithi111:Waiting for deploy to finish 2022-12-14T07:58:33.033 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T07:58:42.116 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T07:58:48.131 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.111 2022-12-14T07:58:51.205 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T07:58:57.216 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.111 2022-12-14T07:59:00.287 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T07:59:06.304 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.111 2022-12-14T07:59:09.375 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T07:59:15.360 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.111 2022-12-14T07:59:18.563 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T07:59:24.450 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.111 2022-12-14T07:59:33.535 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.111 2022-12-14T08:00:24.612 WARNING:teuthology.provision.fog:timed out 2022-12-14T08:00:33.697 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:00:39.551 WARNING:teuthology.provision.fog:timed out 2022-12-14T08:00:42.758 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:00:45.831 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-12-14T08:00:46.080 INFO:teuthology.provision.fog.smithi111:Node is ready 2022-12-14T08:00:46.101 INFO:teuthology.orchestra.run.smithi111.stdout:smithi111.front.sepia.ceph.com 2022-12-14T08:00:46.161 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111 smithi111.front.sepia.ceph.com smithi111 2022-12-14T08:00:46.533 INFO:teuthology.provision.fog.smithi111:Deploy complete! 2022-12-14T08:00:51.843 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:01:00.927 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:01:09.984 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:01:19.071 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:01:28.131 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:01:37.219 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:01:46.307 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:01:55.393 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:02:04.485 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:02:13.567 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:02:22.623 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:02:31.683 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:02:40.771 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:02:49.855 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:02:58.915 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:03:08.007 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:03:17.095 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:03:26.177 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:03:35.267 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:03:44.351 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:03:53.408 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:04:02.463 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:04:11.523 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:04:20.607 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:04:29.663 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:04:38.723 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:04:47.807 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:04:56.863 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:05:05.923 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:05:15.007 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:05:24.063 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:05:33.119 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:05:42.179 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:05:51.263 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:06:00.323 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:06:09.379 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:06:18.463 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:06:27.519 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:06:36.580 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:06:45.663 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:06:54.719 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:07:03.775 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:07:12.831 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:07:21.887 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:07:30.943 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:07:40.003 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:07:49.087 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:07:58.147 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:08:07.231 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:08:16.287 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:08:25.343 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:08:34.399 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:08:43.455 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:08:52.515 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:09:01.599 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:09:10.655 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:09:19.711 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:09:28.799 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:09:37.855 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:09:46.911 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:09:55.971 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:10:05.059 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:10:14.147 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:10:23.231 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:10:32.287 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:10:41.343 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:10:50.403 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:10:59.487 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:11:08.551 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:11:17.635 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:11:26.719 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:11:35.807 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:11:44.863 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:11:53.919 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:12:02.988 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:12:12.067 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:12:21.151 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:12:30.207 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:12:39.263 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:12:48.323 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:12:57.407 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:13:06.463 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:13:15.523 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:13:24.607 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:13:33.663 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:13:42.723 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:13:51.807 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:14:00.899 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:14:09.983 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:14:19.039 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:14:28.095 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2022-12-14T08:14:28.098 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 209, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/lock/ops.py", line 324, in reimage_machines log.info("Node '%s' reimaging is complete", machine) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/provision/__init__.py", line 39, in reimage return obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/provision/fog.py", line 271, in _wait_for_ready while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds 2022-12-14T08:14:28.179 INFO:teuthology.nuke:Checking targets against current locks 2022-12-14T08:14:28.235 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-12-14T08:14:28.262 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-12-14T08:14:28.334 INFO:teuthology.orchestra.console:Power off smithi111 2022-12-14T08:14:28.385 INFO:teuthology.orchestra.console:Power off smithi140 2022-12-14T08:14:36.727 INFO:teuthology.orchestra.console:Power off for smithi111 completed 2022-12-14T08:14:36.878 INFO:teuthology.lock.ops:unlocked: smithi111.front.sepia.ceph.com 2022-12-14T08:14:40.891 INFO:teuthology.orchestra.console:Power off for smithi140 completed 2022-12-14T08:14:41.048 INFO:teuthology.lock.ops:unlocked: smithi140.front.sepia.ceph.com 2022-12-14T08:14:41.273 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/__init__.py", line 63, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 52, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 209, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/lock/ops.py", line 324, in reimage_machines log.info("Node '%s' reimaging is complete", machine) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/provision/__init__.py", line 39, in reimage return obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/provision/fog.py", line 271, in _wait_for_ready while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds