2023-03-11T14:52:03.539 INFO:root:teuthology version: 0.0.1.dev42+ga6a4a7f
2023-03-11T14:52:03.677 INFO:teuthology.lock.ops:Start node 'smithi038.front.sepia.ceph.com' reimaging
2023-03-11T14:52:03.678 INFO:teuthology.lock.ops:Updating [smithi038.front.sepia.ceph.com]: reset os type and version on server
2023-03-11T14:52:03.678 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server
2023-03-11T14:52:03.701 INFO:teuthology.lock.ops:Node 'smithi038.front.sepia.ceph.com' reimaging is complete
2023-03-11T14:52:03.702 INFO:teuthology.lock.ops:Start node 'smithi140.front.sepia.ceph.com' reimaging
2023-03-11T14:52:03.702 INFO:teuthology.lock.ops:Updating [smithi140.front.sepia.ceph.com]: reset os type and version on server
2023-03-11T14:52:03.702 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server
2023-03-11T14:52:03.726 INFO:teuthology.lock.ops:Node 'smithi140.front.sepia.ceph.com' reimaging is complete
2023-03-11T14:52:03.865 INFO:teuthology.provision.fog.smithi038:Scheduling deploy of rhel 8.6
2023-03-11T14:52:03.881 INFO:teuthology.provision.fog.smithi140:Scheduling deploy of rhel 8.6
2023-03-11T14:52:04.102 INFO:teuthology.orchestra.console:Power off smithi038
2023-03-11T14:52:04.118 INFO:teuthology.orchestra.console:Power off smithi140
2023-03-11T14:52:04.251 INFO:teuthology.orchestra.console:Power off for smithi038 completed
2023-03-11T14:52:04.264 INFO:teuthology.orchestra.console:Power off for smithi140 completed
2023-03-11T14:52:04.352 INFO:teuthology.orchestra.console:Power on smithi038
2023-03-11T14:52:04.365 INFO:teuthology.orchestra.console:Power on smithi140
2023-03-11T14:52:08.630 INFO:teuthology.orchestra.console:Power on for smithi140 completed
2023-03-11T14:52:08.731 INFO:teuthology.provision.fog.smithi140:Waiting for deploy to finish
2023-03-11T14:52:12.731 INFO:teuthology.orchestra.console:Power on for smithi038 completed
2023-03-11T14:52:12.832 INFO:teuthology.provision.fog.smithi038:Waiting for deploy to finish
2023-03-11T14:55:02.149 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.38
2023-03-11T14:55:11.237 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.38
2023-03-11T14:55:20.293 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.38
2023-03-11T14:55:28.266 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T14:55:29.353 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.38
2023-03-11T14:55:37.353 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T14:55:46.441 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T14:55:55.525 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T14:56:04.581 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T14:56:35.381 WARNING:teuthology.provision.fog:timed out
2023-03-11T14:56:41.724 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-11T14:56:41.988 INFO:teuthology.provision.fog.smithi038:Node is ready
2023-03-11T14:56:42.047 INFO:teuthology.orchestra.run.smithi038.stdout:smithi038.front.sepia.ceph.com
2023-03-11T14:56:42.107 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38 smithi038.front.sepia.ceph.com smithi038
2023-03-11T14:56:42.460 INFO:teuthology.provision.fog.smithi038:Deploy complete!
2023-03-11T14:57:10.613 WARNING:teuthology.provision.fog:timed out
2023-03-11T14:57:34.981 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T14:57:44.037 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T14:57:53.093 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T14:58:02.149 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T14:58:11.205 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T14:58:20.261 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T14:58:29.321 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T14:58:38.405 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T14:58:47.465 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T14:58:56.549 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T14:59:05.605 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T14:59:14.697 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T14:59:23.781 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T14:59:32.837 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T14:59:41.893 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T14:59:50.949 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:00:00.005 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:00:09.065 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:00:18.153 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:00:27.237 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:00:36.293 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:00:45.353 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:00:54.441 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:01:03.525 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:01:12.581 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:01:21.641 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:01:30.725 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:01:39.781 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:01:48.837 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:01:57.893 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:02:06.951 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:02:16.041 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:02:25.125 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:02:34.181 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:02:43.269 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:02:52.329 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:03:01.413 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:03:10.469 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:03:19.561 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:03:28.645 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:03:37.701 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:03:46.757 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:03:55.845 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:04:04.901 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:04:13.957 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:04:23.013 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:04:32.073 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:04:41.157 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:04:50.213 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:04:59.269 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:05:08.325 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:05:17.381 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:05:26.441 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:05:35.529 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:05:44.613 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:05:53.669 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:06:02.725 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:06:11.785 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:06:20.869 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:06:29.925 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:06:38.985 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:06:48.069 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:06:57.125 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:07:06.181 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:07:15.237 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:07:24.293 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:07:33.349 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:07:42.409 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:07:51.493 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:08:00.549 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:08:09.605 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:08:18.661 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:08:27.717 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:08:36.809 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:08:45.893 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:08:54.949 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:09:04.009 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:09:13.097 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:09:22.181 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:09:31.241 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:09:40.325 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:09:49.417 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:09:58.501 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:10:07.557 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:10:16.613 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:10:25.669 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:10:34.725 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:10:43.785 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:10:52.869 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:11:01.925 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:11:11.013 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:11:20.069 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:11:29.129 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:11:38.217 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-11T15:11:38.218 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/dispatcher/supervisor.py", line 209, in reimage
    reimaged = reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/lock/ops.py", line 324, in reimage_machines
    log.info("Node '%s' reimaging is complete", machine)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/provision/__init__.py", line 39, in reimage
    return obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/provision/fog.py", line 90, in create
    self._wait_for_ready()
  File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/provision/fog.py", line 271, in _wait_for_ready
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/contextutil.py", line 133, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds
2023-03-11T15:11:38.300 INFO:teuthology.nuke:Checking targets against current locks
2023-03-11T15:11:38.359 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-11T15:11:38.379 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-11T15:11:38.439 INFO:teuthology.orchestra.console:Power off smithi038
2023-03-11T15:11:38.458 INFO:teuthology.orchestra.console:Power off smithi140
2023-03-11T15:11:46.837 INFO:teuthology.orchestra.console:Power off for smithi038 completed
2023-03-11T15:11:46.859 INFO:teuthology.orchestra.console:Power off for smithi140 completed
2023-03-11T15:11:46.998 INFO:teuthology.lock.ops:unlocked: smithi038.front.sepia.ceph.com
2023-03-11T15:11:47.006 INFO:teuthology.lock.ops:unlocked: smithi140.front.sepia.ceph.com
2023-03-11T15:11:47.242 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/dispatcher/__init__.py", line 63, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/dispatcher/supervisor.py", line 52, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/dispatcher/supervisor.py", line 209, in reimage
    reimaged = reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/lock/ops.py", line 324, in reimage_machines
    log.info("Node '%s' reimaging is complete", machine)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/provision/__init__.py", line 39, in reimage
    return obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/provision/fog.py", line 90, in create
    self._wait_for_ready()
  File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/provision/fog.py", line 271, in _wait_for_ready
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_teuthology_a6a4a7f010ae6b3f7fc2aef91377d4a6bee6de40/teuthology/contextutil.py", line 133, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds