2023-03-09T11:12:55.135 INFO:root:teuthology version: 0.0.1.dev51+ga2dfb05
2023-03-09T11:12:55.290 INFO:teuthology.lock.ops:Start node 'smithi140.front.sepia.ceph.com' reimaging
2023-03-09T11:12:55.292 INFO:teuthology.lock.ops:Updating [smithi140.front.sepia.ceph.com]: reset os type and version on server
2023-03-09T11:12:55.292 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server
2023-03-09T11:12:55.316 INFO:teuthology.lock.ops:Node 'smithi140.front.sepia.ceph.com' reimaging is complete
2023-03-09T11:12:55.317 INFO:teuthology.lock.ops:Start node 'smithi156.front.sepia.ceph.com' reimaging
2023-03-09T11:12:55.317 INFO:teuthology.lock.ops:Updating [smithi156.front.sepia.ceph.com]: reset os type and version on server
2023-03-09T11:12:55.318 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server
2023-03-09T11:12:55.348 INFO:teuthology.lock.ops:Node 'smithi156.front.sepia.ceph.com' reimaging is complete
2023-03-09T11:12:55.478 INFO:teuthology.provision.fog.smithi140:Scheduling deploy of rhel 8.6
2023-03-09T11:12:55.507 INFO:teuthology.provision.fog.smithi156:Scheduling deploy of rhel 8.6
2023-03-09T11:12:55.716 INFO:teuthology.orchestra.console:Power off smithi140
2023-03-09T11:12:55.763 INFO:teuthology.orchestra.console:Power off smithi156
2023-03-09T11:13:04.105 INFO:teuthology.orchestra.console:Power off for smithi140 completed
2023-03-09T11:13:04.138 INFO:teuthology.orchestra.console:Power off for smithi156 completed
2023-03-09T11:13:04.207 INFO:teuthology.orchestra.console:Power on smithi140
2023-03-09T11:13:04.239 INFO:teuthology.orchestra.console:Power on smithi156
2023-03-09T11:13:08.497 INFO:teuthology.orchestra.console:Power on for smithi156 completed
2023-03-09T11:13:08.598 INFO:teuthology.provision.fog.smithi156:Waiting for deploy to finish
2023-03-09T11:13:12.584 INFO:teuthology.orchestra.console:Power on for smithi140 completed
2023-03-09T11:13:12.686 INFO:teuthology.provision.fog.smithi140:Waiting for deploy to finish
2023-03-09T11:16:17.033 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:16:26.117 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:16:28.037 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.156
2023-03-09T11:16:35.173 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:16:37.097 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.156
2023-03-09T11:16:44.229 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:16:46.181 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.156
2023-03-09T11:16:53.285 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:16:55.237 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.156
2023-03-09T11:17:02.343 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:17:04.293 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.156
2023-03-09T11:18:08.405 WARNING:teuthology.provision.fog:timed out
2023-03-09T11:18:10.297 WARNING:teuthology.provision.fog:timed out
2023-03-09T11:18:16.604 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-09T11:18:16.915 INFO:teuthology.provision.fog.smithi156:Node is ready
2023-03-09T11:18:16.934 INFO:teuthology.orchestra.run.smithi156.stdout:smithi156.front.sepia.ceph.com
2023-03-09T11:18:16.990 INFO:teuthology.orchestra.run.smithi156.stdout:172.21.15.156 smithi156.front.sepia.ceph.com smithi156
2023-03-09T11:18:17.354 INFO:teuthology.provision.fog.smithi156:Deploy complete!
2023-03-09T11:18:17.477 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:18:26.533 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:18:35.594 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:18:44.677 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:18:53.733 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:19:02.789 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:19:11.845 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:19:20.901 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:19:29.957 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:19:39.017 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:19:48.105 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:19:57.189 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:20:06.245 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:20:15.333 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:20:24.389 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:20:33.445 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:20:42.506 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:20:51.593 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:21:00.677 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:21:09.733 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:21:18.789 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:21:27.849 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:21:36.933 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:21:45.993 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:21:55.077 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:22:04.137 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:22:13.221 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:22:22.281 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:22:31.369 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:22:40.453 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:22:49.509 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:22:58.565 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:23:07.625 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:23:16.709 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:23:25.765 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:23:34.821 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:23:43.881 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:23:52.969 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:24:02.053 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:24:11.109 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:24:20.165 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:24:29.221 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:24:38.281 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:24:47.369 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:24:56.453 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:25:05.509 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:25:14.565 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:25:23.621 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:25:32.677 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:25:41.733 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:25:50.821 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:25:59.877 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:26:08.933 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:26:17.989 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:26:27.045 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:26:36.101 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:26:45.157 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:26:54.217 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:27:03.301 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:27:12.357 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:27:21.413 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:27:30.501 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:27:39.589 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:27:48.646 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:27:57.701 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:28:06.757 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:28:15.845 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:28:24.901 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:28:33.957 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:28:43.013 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:28:52.070 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:29:01.157 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:29:10.213 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:29:19.269 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:29:28.325 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:29:37.413 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:29:46.469 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:29:55.529 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:30:04.613 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:30:13.669 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:30:22.725 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:30:31.813 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:30:40.873 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:30:49.961 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:30:59.045 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:31:08.101 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:31:17.157 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:31:26.213 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:31:35.305 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:31:44.389 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:31:53.445 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:32:02.506 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:32:11.593 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T11:32:11.594 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/dispatcher/supervisor.py", line 209, in reimage
    reimaged = reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/lock/ops.py", line 324, in reimage_machines
    log.info("Node '%s' reimaging is complete", machine)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/provision/__init__.py", line 39, in reimage
    return obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/provision/fog.py", line 90, in create
    self._wait_for_ready()
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/provision/fog.py", line 271, in _wait_for_ready
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/contextutil.py", line 135, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds
2023-03-09T11:32:11.681 INFO:teuthology.nuke:Checking targets against current locks
2023-03-09T11:32:11.741 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-09T11:32:11.765 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-09T11:32:11.835 INFO:teuthology.orchestra.console:Power off smithi140
2023-03-09T11:32:11.887 INFO:teuthology.orchestra.console:Power off smithi156
2023-03-09T11:32:20.229 INFO:teuthology.orchestra.console:Power off for smithi140 completed
2023-03-09T11:32:20.270 INFO:teuthology.orchestra.console:Power off for smithi156 completed
2023-03-09T11:32:20.387 INFO:teuthology.lock.ops:unlocked: smithi140.front.sepia.ceph.com
2023-03-09T11:32:20.426 INFO:teuthology.lock.ops:unlocked: smithi156.front.sepia.ceph.com
2023-03-09T11:32:20.668 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/dispatcher/__init__.py", line 63, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/dispatcher/supervisor.py", line 52, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/dispatcher/supervisor.py", line 209, in reimage
    reimaged = reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/lock/ops.py", line 324, in reimage_machines
    log.info("Node '%s' reimaging is complete", machine)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/provision/__init__.py", line 39, in reimage
    return obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/provision/fog.py", line 90, in create
    self._wait_for_ready()
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/provision/fog.py", line 271, in _wait_for_ready
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/contextutil.py", line 135, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds