2023-01-17T14:37:48.914 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2023-01-17T14:37:49.039 INFO:teuthology.lock.ops:Start node 'smithi140.front.sepia.ceph.com' reimaging 2023-01-17T14:37:49.039 INFO:teuthology.lock.ops:Updating [smithi140.front.sepia.ceph.com]: reset os type and version on server 2023-01-17T14:37:49.039 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server 2023-01-17T14:37:49.054 INFO:teuthology.lock.ops:Node 'smithi140.front.sepia.ceph.com' reimaging is complete 2023-01-17T14:37:49.054 INFO:teuthology.lock.ops:Start node 'smithi149.front.sepia.ceph.com' reimaging 2023-01-17T14:37:49.054 INFO:teuthology.lock.ops:Updating [smithi149.front.sepia.ceph.com]: reset os type and version on server 2023-01-17T14:37:49.055 INFO:teuthology.lock.ops:Updating smithi149.front.sepia.ceph.com on lock server 2023-01-17T14:37:49.070 INFO:teuthology.lock.ops:Node 'smithi149.front.sepia.ceph.com' reimaging is complete 2023-01-17T14:37:49.207 INFO:teuthology.provision.fog.smithi140:Scheduling deploy of rhel 8.6 2023-01-17T14:37:49.218 INFO:teuthology.provision.fog.smithi149:Scheduling deploy of rhel 8.6 2023-01-17T14:37:49.443 INFO:teuthology.orchestra.console:Power off smithi149 2023-01-17T14:37:49.456 INFO:teuthology.orchestra.console:Power off smithi140 2023-01-17T14:37:57.813 INFO:teuthology.orchestra.console:Power off for smithi149 completed 2023-01-17T14:37:57.820 INFO:teuthology.orchestra.console:Power off for smithi140 completed 2023-01-17T14:37:57.914 INFO:teuthology.orchestra.console:Power on smithi149 2023-01-17T14:37:57.924 INFO:teuthology.orchestra.console:Power on smithi140 2023-01-17T14:38:06.283 INFO:teuthology.orchestra.console:Power on for smithi149 completed 2023-01-17T14:38:06.333 INFO:teuthology.orchestra.console:Power on for smithi140 completed 2023-01-17T14:38:06.384 INFO:teuthology.provision.fog.smithi149:Waiting for deploy to finish 2023-01-17T14:38:06.434 INFO:teuthology.provision.fog.smithi140:Waiting for deploy to finish 2023-01-17T14:41:10.761 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:41:10.762 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.149 2023-01-17T14:41:19.821 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:41:19.831 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.149 2023-01-17T14:41:28.905 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.149 2023-01-17T14:41:29.108 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:41:38.189 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.149 2023-01-17T14:41:38.254 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:41:47.337 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.149 2023-01-17T14:41:47.337 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:42:53.401 WARNING:teuthology.provision.fog:timed out 2023-01-17T14:42:53.401 WARNING:teuthology.provision.fog:timed out 2023-01-17T14:42:59.960 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-01-17T14:43:00.241 INFO:teuthology.provision.fog.smithi149:Node is ready 2023-01-17T14:43:00.260 INFO:teuthology.orchestra.run.smithi149.stdout:smithi149.front.sepia.ceph.com 2023-01-17T14:43:00.318 INFO:teuthology.orchestra.run.smithi149.stdout:172.21.15.149 smithi149.front.sepia.ceph.com smithi149 2023-01-17T14:43:00.666 INFO:teuthology.provision.fog.smithi149:Deploy complete! 2023-01-17T14:43:17.741 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:43:26.825 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:43:35.881 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:43:44.937 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:43:54.025 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:44:03.081 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:44:12.137 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:44:21.229 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:44:30.317 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:44:39.405 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:44:48.489 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:44:57.549 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:45:06.633 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:45:15.689 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:45:24.745 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:45:33.801 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:45:42.857 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:45:51.917 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:46:01.001 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:46:10.057 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:46:19.117 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:46:28.205 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:46:37.293 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:46:46.377 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:46:55.437 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:47:04.521 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:47:13.577 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:47:22.636 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:47:31.721 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:47:41.417 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:47:50.637 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:47:59.721 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:48:08.777 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:48:18.345 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:48:27.401 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:48:36.685 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:48:45.769 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:48:54.829 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:49:03.913 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:49:12.973 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:49:22.061 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:49:31.145 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:49:40.201 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:49:49.289 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:49:58.409 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:50:07.785 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:50:17.069 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:50:26.349 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:50:35.721 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:50:44.781 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:50:53.869 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:51:02.957 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:51:12.041 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:51:21.101 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:51:30.189 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:51:39.369 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:51:48.525 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:51:57.609 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:52:06.669 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:52:15.753 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:52:24.874 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:52:33.997 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:52:43.081 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:52:52.169 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:53:01.321 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:53:10.377 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:53:19.437 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:53:28.525 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:53:37.609 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:53:46.669 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:53:55.753 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:54:04.809 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:54:13.865 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:54:22.921 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:54:31.981 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:54:41.065 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:54:50.221 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:54:59.337 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:55:08.393 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:55:17.641 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:55:26.825 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:55:35.885 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:55:45.385 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:55:54.441 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:56:03.497 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:56:12.557 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:56:22.473 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:56:31.725 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:56:40.877 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:56:50.089 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:56:59.145 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:57:08.393 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:57:17.449 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:57:26.537 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-01-17T14:57:26.538 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 209, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/lock/ops.py", line 324, in reimage_machines log.info("Node '%s' reimaging is complete", machine) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/provision/__init__.py", line 39, in reimage return obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/provision/fog.py", line 271, in _wait_for_ready while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds 2023-01-17T14:57:26.610 INFO:teuthology.nuke:Checking targets against current locks 2023-01-17T14:57:26.658 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-17T14:57:26.678 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-17T14:57:26.733 INFO:teuthology.orchestra.console:Power off smithi140 2023-01-17T14:57:26.770 INFO:teuthology.orchestra.console:Power off smithi149 2023-01-17T14:57:35.135 INFO:teuthology.orchestra.console:Power off for smithi149 completed 2023-01-17T14:57:35.285 INFO:teuthology.lock.ops:unlocked: smithi149.front.sepia.ceph.com 2023-01-17T14:57:39.214 INFO:teuthology.orchestra.console:Power off for smithi140 completed 2023-01-17T14:57:39.359 INFO:teuthology.lock.ops:unlocked: smithi140.front.sepia.ceph.com 2023-01-17T14:57:44.235 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/__init__.py", line 63, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 52, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 209, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/lock/ops.py", line 324, in reimage_machines log.info("Node '%s' reimaging is complete", machine) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/provision/__init__.py", line 39, in reimage return obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/provision/fog.py", line 271, in _wait_for_ready while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds