2023-03-07T20:36:51.201 INFO:root:teuthology version: 0.0.1.dev51+ga2dfb05
2023-03-07T20:36:51.344 INFO:teuthology.lock.ops:Start node 'smithi140.front.sepia.ceph.com' reimaging
2023-03-07T20:36:51.345 INFO:teuthology.lock.ops:Updating [smithi140.front.sepia.ceph.com]: reset os type and version on server
2023-03-07T20:36:51.345 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server
2023-03-07T20:36:51.371 INFO:teuthology.lock.ops:Node 'smithi140.front.sepia.ceph.com' reimaging is complete
2023-03-07T20:36:51.371 INFO:teuthology.lock.ops:Start node 'smithi188.front.sepia.ceph.com' reimaging
2023-03-07T20:36:51.371 INFO:teuthology.lock.ops:Updating [smithi188.front.sepia.ceph.com]: reset os type and version on server
2023-03-07T20:36:51.372 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server
2023-03-07T20:36:51.395 INFO:teuthology.lock.ops:Node 'smithi188.front.sepia.ceph.com' reimaging is complete
2023-03-07T20:36:51.530 INFO:teuthology.provision.fog.smithi140:Scheduling deploy of rhel 8.6
2023-03-07T20:36:51.539 INFO:teuthology.provision.fog.smithi188:Scheduling deploy of rhel 8.6
2023-03-07T20:36:51.786 INFO:teuthology.orchestra.console:Power off smithi140
2023-03-07T20:36:51.799 INFO:teuthology.orchestra.console:Power off smithi188
2023-03-07T20:36:51.926 INFO:teuthology.orchestra.console:Power off for smithi140 completed
2023-03-07T20:36:52.027 INFO:teuthology.orchestra.console:Power on smithi140
2023-03-07T20:37:00.164 INFO:teuthology.orchestra.console:Power off for smithi188 completed
2023-03-07T20:37:00.266 INFO:teuthology.orchestra.console:Power on smithi188
2023-03-07T20:37:00.395 INFO:teuthology.orchestra.console:Power on for smithi140 completed
2023-03-07T20:37:00.496 INFO:teuthology.provision.fog.smithi140:Waiting for deploy to finish
2023-03-07T20:37:08.631 INFO:teuthology.orchestra.console:Power on for smithi188 completed
2023-03-07T20:37:08.732 INFO:teuthology.provision.fog.smithi188:Waiting for deploy to finish
2023-03-07T20:40:50.949 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:41:00.005 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:41:09.061 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:41:18.117 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:41:26.487 ERROR:teuthology.orchestra.connection:Error authenticating with smithi188.front.sepia.ceph.com: Authentication failed.
2023-03-07T20:41:27.177 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:41:36.229 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:42:27.491 WARNING:teuthology.provision.fog:timed out
2023-03-07T20:42:42.231 WARNING:teuthology.provision.fog:timed out
2023-03-07T20:42:52.041 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.188
2023-03-07T20:43:01.093 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.188
2023-03-07T20:43:06.629 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:43:08.435 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-07T20:43:08.731 INFO:teuthology.provision.fog.smithi188:Node is ready
2023-03-07T20:43:08.747 INFO:teuthology.orchestra.run.smithi188.stdout:smithi022
2023-03-07T20:43:08.802 INFO:teuthology.orchestra.run.smithi188.stdout:172.21.15.22 smithi022.front.sepia.ceph.com smithi022
2023-03-07T20:43:09.146 INFO:teuthology.provision.fog.smithi188:Deploy complete!
2023-03-07T20:43:15.685 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:43:24.741 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:43:33.797 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:43:42.853 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:43:51.909 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:44:00.965 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:44:10.025 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:44:19.077 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:44:28.133 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:44:37.189 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:44:46.245 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:44:55.301 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:45:04.361 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:45:13.417 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:45:22.501 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:45:31.561 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:45:40.645 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:45:49.701 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:45:58.757 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:46:07.817 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:46:16.869 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:46:25.925 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:46:34.985 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:46:44.037 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:46:53.093 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:47:02.149 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:47:11.205 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:47:20.261 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:47:29.321 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:47:38.373 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:47:47.429 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:47:56.485 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:48:05.541 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:48:14.601 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:48:23.653 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:48:32.709 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:48:41.765 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:48:50.821 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:48:59.877 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:49:08.933 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:49:17.989 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:49:27.045 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:49:36.105 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:49:45.161 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:49:54.217 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:50:03.273 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:50:12.329 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:50:21.385 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:50:30.441 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:50:39.495 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:50:48.549 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:50:57.609 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:51:06.665 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:51:15.717 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:51:24.777 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:51:33.829 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:51:42.885 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:51:51.941 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:52:01.001 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:52:10.053 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:52:19.109 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:52:28.169 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:52:37.225 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:52:46.277 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:52:55.333 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:53:04.389 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:53:13.445 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:53:22.505 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:53:31.563 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:53:40.645 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:53:49.701 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:53:58.761 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:54:07.813 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:54:16.869 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:54:25.925 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:54:34.981 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:54:44.037 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:54:53.093 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:55:02.153 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:55:11.205 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:55:20.261 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:55:29.317 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:55:38.377 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:55:47.465 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:55:56.517 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:56:05.573 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:56:14.629 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:56:23.685 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:56:32.745 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:56:41.797 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:56:50.853 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:56:59.909 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-07T20:56:59.911 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/dispatcher/supervisor.py", line 209, in reimage
    reimaged = reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/lock/ops.py", line 324, in reimage_machines
    log.info("Node '%s' reimaging is complete", machine)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/provision/__init__.py", line 39, in reimage
    return obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/provision/fog.py", line 90, in create
    self._wait_for_ready()
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/provision/fog.py", line 271, in _wait_for_ready
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/contextutil.py", line 135, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds
2023-03-07T20:56:59.976 INFO:teuthology.nuke:Checking targets against current locks
2023-03-07T20:57:00.032 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-07T20:57:00.052 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-07T20:57:00.106 INFO:teuthology.orchestra.console:Power off smithi140
2023-03-07T20:57:00.126 INFO:teuthology.orchestra.console:Power off smithi188
2023-03-07T20:57:12.590 INFO:teuthology.orchestra.console:Power off for smithi140 completed
2023-03-07T20:57:12.596 INFO:teuthology.orchestra.console:Power off for smithi188 completed
2023-03-07T20:57:12.748 INFO:teuthology.lock.ops:unlocked: smithi188.front.sepia.ceph.com
2023-03-07T20:57:12.756 INFO:teuthology.lock.ops:unlocked: smithi140.front.sepia.ceph.com
2023-03-07T20:57:12.992 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/dispatcher/__init__.py", line 63, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/dispatcher/supervisor.py", line 52, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/dispatcher/supervisor.py", line 209, in reimage
    reimaged = reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/lock/ops.py", line 324, in reimage_machines
    log.info("Node '%s' reimaging is complete", machine)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/provision/__init__.py", line 39, in reimage
    return obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/provision/fog.py", line 90, in create
    self._wait_for_ready()
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/provision/fog.py", line 271, in _wait_for_ready
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/contextutil.py", line 135, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds