2023-05-11T19:21:44.961 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-11T19:21:45.114 INFO:teuthology.lock.ops:Start node 'smithi063.front.sepia.ceph.com' reimaging 2023-05-11T19:21:45.115 INFO:teuthology.lock.ops:Updating [smithi063.front.sepia.ceph.com]: reset os type and version on server 2023-05-11T19:21:45.116 INFO:teuthology.lock.ops:Updating smithi063.front.sepia.ceph.com on lock server 2023-05-11T19:21:45.139 INFO:teuthology.lock.ops:Node 'smithi063.front.sepia.ceph.com' reimaging is complete 2023-05-11T19:21:45.140 INFO:teuthology.lock.ops:Start node 'smithi169.front.sepia.ceph.com' reimaging 2023-05-11T19:21:45.140 INFO:teuthology.lock.ops:Updating [smithi169.front.sepia.ceph.com]: reset os type and version on server 2023-05-11T19:21:45.141 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server 2023-05-11T19:22:00.171 INFO:teuthology.lock.ops:Node 'smithi169.front.sepia.ceph.com' reimaging is complete 2023-05-11T19:22:25.303 INFO:teuthology.provision.fog.smithi063:Scheduling deploy of centos 8.stream 2023-05-11T19:22:25.361 INFO:teuthology.provision.fog.smithi169:Scheduling deploy of centos 8.stream 2023-05-11T19:22:25.596 INFO:teuthology.orchestra.console:Power off smithi063 2023-05-11T19:22:25.734 INFO:teuthology.orchestra.console:Power off smithi169 2023-05-11T19:22:25.743 INFO:teuthology.orchestra.console:Power off for smithi063 completed 2023-05-11T19:22:25.844 INFO:teuthology.orchestra.console:Power on smithi063 2023-05-11T19:22:25.879 INFO:teuthology.orchestra.console:Power off for smithi169 completed 2023-05-11T19:22:25.981 INFO:teuthology.orchestra.console:Power on smithi169 2023-05-11T19:22:40.463 INFO:teuthology.orchestra.console:Power on for smithi169 completed 2023-05-11T19:22:40.564 INFO:teuthology.provision.fog.smithi169:Waiting for deploy to finish 2023-05-11T19:22:44.437 INFO:teuthology.orchestra.console:Power on for smithi063 completed 2023-05-11T19:22:44.538 INFO:teuthology.provision.fog.smithi063:Waiting for deploy to finish 2023-05-11T19:23:17.940 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:23:33.953 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:23:42.064 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.63 2023-05-11T19:23:43.024 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:24:16.628 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:24:28.068 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-05-11T19:24:28.337 INFO:teuthology.provision.fog.smithi063:Node is ready 2023-05-11T19:24:28.355 INFO:teuthology.orchestra.run.smithi063.stdout:smithi063.front.sepia.ceph.com 2023-05-11T19:24:28.418 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63 smithi063.front.sepia.ceph.com smithi063 2023-05-11T19:24:28.765 INFO:teuthology.provision.fog.smithi063:Deploy complete! 2023-05-11T19:24:30.708 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:24:39.792 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:24:48.852 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:25:02.928 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:25:13.296 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:25:27.376 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:25:41.456 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:25:55.536 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:26:09.616 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:26:23.696 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:26:35.728 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:26:44.784 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:26:58.864 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:27:12.944 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:27:27.024 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:27:41.104 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:27:55.184 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:28:04.240 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:28:18.324 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:28:32.400 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:28:46.480 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:28:55.540 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:29:09.616 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:29:18.676 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:29:27.760 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:29:36.820 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:29:50.896 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:29:59.952 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:30:14.032 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:30:23.120 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:30:32.180 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:30:46.256 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:30:56.752 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:31:05.840 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:31:14.896 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:31:28.976 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:31:38.032 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:31:52.112 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:32:02.160 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:32:11.248 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:32:20.336 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:32:34.416 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:32:48.496 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:32:59.056 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:33:08.116 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:33:17.200 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:33:31.284 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:33:45.364 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:33:59.444 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:34:13.524 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:34:22.608 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:34:31.664 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:34:45.744 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:34:59.824 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:35:13.904 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:35:22.960 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:35:37.040 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:35:46.100 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:36:00.176 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:36:10.576 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:36:24.660 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:36:33.744 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:36:42.832 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:36:56.912 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:37:10.992 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:37:25.076 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:37:39.152 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:37:53.232 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:38:07.316 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:38:21.392 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:38:30.448 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:38:39.540 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:38:48.624 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:39:02.708 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:39:16.788 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:39:30.864 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:39:41.328 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:39:55.408 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:40:09.492 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:40:20.912 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:40:34.992 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:40:49.072 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:41:03.156 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:41:17.236 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:41:31.312 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:41:40.368 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:41:54.448 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:42:03.508 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:42:12.592 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:42:21.648 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:42:35.728 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:42:44.784 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:42:53.872 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:43:02.964 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:43:17.044 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:43:31.120 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-05-11T19:43:31.122 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/dispatcher/supervisor.py", line 226, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/lock/ops.py", line 324, in reimage_machines log.info("Node '%s' reimaging is complete", machine) File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/provision/__init__.py", line 39, in reimage return obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/provision/fog.py", line 271, in _wait_for_ready while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/contextutil.py", line 135, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds 2023-05-11T19:43:36.214 INFO:teuthology.nuke:Checking targets against current locks 2023-05-11T19:43:46.285 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-05-11T19:43:46.316 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-05-11T19:43:46.371 INFO:teuthology.orchestra.console:Power off smithi063 2023-05-11T19:43:46.389 INFO:teuthology.orchestra.console:Power off smithi169 2023-05-11T19:44:10.096 INFO:teuthology.orchestra.console:Power off for smithi169 completed 2023-05-11T19:44:20.271 INFO:teuthology.lock.ops:unlocked: smithi169.front.sepia.ceph.com 2023-05-11T19:44:20.345 INFO:teuthology.orchestra.console:Power off for smithi063 completed 2023-05-11T19:44:30.502 INFO:teuthology.lock.ops:unlocked: smithi063.front.sepia.ceph.com 2023-05-11T19:44:40.753 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/dispatcher/supervisor.py", line 56, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/dispatcher/supervisor.py", line 226, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/lock/ops.py", line 324, in reimage_machines log.info("Node '%s' reimaging is complete", machine) File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/provision/__init__.py", line 39, in reimage return obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/provision/fog.py", line 271, in _wait_for_ready while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/contextutil.py", line 135, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds