2023-05-05T14:24:21.217 INFO:root:teuthology version: 0.0.1.dev98+g69caab8
2023-05-05T14:24:21.470 INFO:teuthology.lock.ops:Start node 'smithi031.front.sepia.ceph.com' reimaging
2023-05-05T14:24:21.472 INFO:teuthology.lock.ops:Updating [smithi031.front.sepia.ceph.com]: reset os type and version on server
2023-05-05T14:24:21.472 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server
2023-05-05T14:24:21.497 INFO:teuthology.lock.ops:Node 'smithi031.front.sepia.ceph.com' reimaging is complete
2023-05-05T14:24:21.497 INFO:teuthology.lock.ops:Start node 'smithi103.front.sepia.ceph.com' reimaging
2023-05-05T14:24:21.498 INFO:teuthology.lock.ops:Updating [smithi103.front.sepia.ceph.com]: reset os type and version on server
2023-05-05T14:24:21.498 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server
2023-05-05T14:24:21.521 INFO:teuthology.lock.ops:Node 'smithi103.front.sepia.ceph.com' reimaging is complete
2023-05-05T14:24:21.521 INFO:teuthology.lock.ops:Start node 'smithi120.front.sepia.ceph.com' reimaging
2023-05-05T14:24:21.521 INFO:teuthology.lock.ops:Updating [smithi120.front.sepia.ceph.com]: reset os type and version on server
2023-05-05T14:24:21.522 INFO:teuthology.lock.ops:Updating smithi120.front.sepia.ceph.com on lock server
2023-05-05T14:24:21.546 INFO:teuthology.lock.ops:Node 'smithi120.front.sepia.ceph.com' reimaging is complete
2023-05-05T14:24:21.547 INFO:teuthology.lock.ops:Start node 'smithi136.front.sepia.ceph.com' reimaging
2023-05-05T14:24:21.547 INFO:teuthology.lock.ops:Updating [smithi136.front.sepia.ceph.com]: reset os type and version on server
2023-05-05T14:24:21.548 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server
2023-05-05T14:24:21.570 INFO:teuthology.lock.ops:Node 'smithi136.front.sepia.ceph.com' reimaging is complete
2023-05-05T14:24:21.571 INFO:teuthology.lock.ops:Start node 'smithi190.front.sepia.ceph.com' reimaging
2023-05-05T14:24:21.571 INFO:teuthology.lock.ops:Updating [smithi190.front.sepia.ceph.com]: reset os type and version on server
2023-05-05T14:24:21.571 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server
2023-05-05T14:24:21.595 INFO:teuthology.lock.ops:Node 'smithi190.front.sepia.ceph.com' reimaging is complete
2023-05-05T14:24:21.667 INFO:teuthology.provision.fog.smithi031:Scheduling deploy of ubuntu 20.04
2023-05-05T14:24:21.682 INFO:teuthology.provision.fog.smithi103:Scheduling deploy of ubuntu 20.04
2023-05-05T14:24:21.726 INFO:teuthology.provision.fog.smithi120:Scheduling deploy of ubuntu 20.04
2023-05-05T14:24:21.729 INFO:teuthology.provision.fog.smithi136:Scheduling deploy of ubuntu 20.04
2023-05-05T14:24:21.776 INFO:teuthology.provision.fog.smithi190:Scheduling deploy of ubuntu 20.04
2023-05-05T14:24:22.262 INFO:teuthology.orchestra.console:Power off smithi136
2023-05-05T14:24:22.281 INFO:teuthology.orchestra.console:Power off smithi103
2023-05-05T14:24:22.340 INFO:teuthology.orchestra.console:Power off smithi120
2023-05-05T14:24:22.353 INFO:teuthology.orchestra.console:Power off smithi031
2023-05-05T14:24:22.415 INFO:teuthology.orchestra.console:Power off for smithi136 completed
2023-05-05T14:24:22.432 INFO:teuthology.orchestra.console:Power off smithi190
2023-05-05T14:24:22.441 INFO:teuthology.orchestra.console:Power off for smithi103 completed
2023-05-05T14:24:22.486 INFO:teuthology.orchestra.console:Power off for smithi120 completed
2023-05-05T14:24:22.501 INFO:teuthology.orchestra.console:Power off for smithi031 completed
2023-05-05T14:24:22.517 INFO:teuthology.orchestra.console:Power on smithi136
2023-05-05T14:24:22.546 INFO:teuthology.orchestra.console:Power on smithi103
2023-05-05T14:24:22.577 INFO:teuthology.orchestra.console:Power off for smithi190 completed
2023-05-05T14:24:22.588 INFO:teuthology.orchestra.console:Power on smithi120
2023-05-05T14:24:22.602 INFO:teuthology.orchestra.console:Power on smithi031
2023-05-05T14:24:22.680 INFO:teuthology.orchestra.console:Power on smithi190
2023-05-05T14:24:26.844 INFO:teuthology.orchestra.console:Power on for smithi120 completed
2023-05-05T14:24:26.856 INFO:teuthology.orchestra.console:Power on for smithi031 completed
2023-05-05T14:24:26.933 INFO:teuthology.orchestra.console:Power on for smithi190 completed
2023-05-05T14:24:26.946 INFO:teuthology.provision.fog.smithi120:Waiting for deploy to finish
2023-05-05T14:24:26.958 INFO:teuthology.provision.fog.smithi031:Waiting for deploy to finish
2023-05-05T14:24:27.035 INFO:teuthology.provision.fog.smithi190:Waiting for deploy to finish
2023-05-05T14:24:30.889 INFO:teuthology.orchestra.console:Power on for smithi136 completed
2023-05-05T14:24:30.917 INFO:teuthology.orchestra.console:Power on for smithi103 completed
2023-05-05T14:24:30.991 INFO:teuthology.provision.fog.smithi136:Waiting for deploy to finish
2023-05-05T14:24:31.018 INFO:teuthology.provision.fog.smithi103:Waiting for deploy to finish
2023-05-05T14:28:03.461 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.31
2023-05-05T14:28:12.517 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.31
2023-05-05T14:28:21.573 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.31
2023-05-05T14:28:30.629 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.31
2023-05-05T14:29:36.634 WARNING:teuthology.provision.fog:timed out
2023-05-05T14:29:42.846 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-05-05T14:29:43.486 INFO:teuthology.provision.fog.smithi031:Node is ready
2023-05-05T14:29:43.492 INFO:teuthology.orchestra.run.smithi031.stdout:smithi031.front.sepia.ceph.com
2023-05-05T14:29:43.499 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31 smithi031.front.sepia.ceph.com smithi031
2023-05-05T14:29:43.669 INFO:teuthology.provision.fog.smithi031:Deploy complete!
2023-05-05T14:31:52.325 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.120
2023-05-05T14:32:01.381 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.120
2023-05-05T14:32:10.437 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.120
2023-05-05T14:32:19.493 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.120
2023-05-05T14:32:28.549 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.120
2023-05-05T14:32:37.605 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.120
2023-05-05T14:33:43.613 WARNING:teuthology.provision.fog:timed out
2023-05-05T14:33:49.821 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-05-05T14:33:50.308 INFO:teuthology.provision.fog.smithi120:Node is ready
2023-05-05T14:33:50.314 INFO:teuthology.orchestra.run.smithi120.stdout:smithi120.front.sepia.ceph.com
2023-05-05T14:33:50.321 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.15.120 smithi120.front.sepia.ceph.com smithi120
2023-05-05T14:33:50.484 INFO:teuthology.provision.fog.smithi120:Deploy complete!
2023-05-05T14:37:11.045 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.190
2023-05-05T14:37:20.105 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.190
2023-05-05T14:37:29.189 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.190
2023-05-05T14:37:38.249 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.190
2023-05-05T14:37:47.337 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.190
2023-05-05T14:37:56.421 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.190
2023-05-05T14:38:34.660 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-05-05T14:38:55.631 INFO:teuthology.provision.fog.smithi190:Node is ready
2023-05-05T14:38:55.638 INFO:teuthology.orchestra.run.smithi190.stdout:smithi190.front.sepia.ceph.com
2023-05-05T14:38:55.687 INFO:teuthology.orchestra.run.smithi190.stdout:172.21.15.190 smithi190.front.sepia.ceph.com smithi190
2023-05-05T14:38:55.867 INFO:teuthology.provision.fog.smithi190:Deploy complete!
2023-05-05T14:39:27.973 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/dispatcher/supervisor.py", line 226, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/lock/ops.py", line 324, in reimage_machines
    log.info("Node '%s' reimaging is complete", machine)
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/provision/__init__.py", line 39, in reimage
    return obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/provision/fog.py", line 86, in create
    self.wait_for_deploy_task(task_id)
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/provision/fog.py", line 255, in wait_for_deploy_task
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/contextutil.py", line 135, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: reached maximum tries (60) after waiting for 900 seconds
2023-05-05T14:39:28.065 INFO:teuthology.nuke:Checking targets against current locks
2023-05-05T14:39:28.115 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-05-05T14:39:28.149 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-05-05T14:39:28.174 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-05-05T14:39:28.189 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-05-05T14:39:28.207 INFO:teuthology.orchestra.console:Power off smithi031
2023-05-05T14:39:28.224 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-05-05T14:39:28.257 INFO:teuthology.orchestra.console:Power off smithi103
2023-05-05T14:39:28.280 INFO:teuthology.orchestra.console:Power off smithi120
2023-05-05T14:39:28.328 INFO:teuthology.orchestra.console:Power off smithi136
2023-05-05T14:39:28.370 INFO:teuthology.orchestra.console:Power off smithi190
2023-05-05T14:39:36.644 INFO:teuthology.orchestra.console:Power off for smithi103 completed
2023-05-05T14:39:36.799 INFO:teuthology.lock.ops:unlocked: smithi103.front.sepia.ceph.com
2023-05-05T14:39:40.740 INFO:teuthology.orchestra.console:Power off for smithi031 completed
2023-05-05T14:39:40.783 INFO:teuthology.orchestra.console:Power off for smithi120 completed
2023-05-05T14:39:40.817 INFO:teuthology.orchestra.console:Power off for smithi136 completed
2023-05-05T14:39:40.852 INFO:teuthology.orchestra.console:Power off for smithi190 completed
2023-05-05T14:39:40.922 INFO:teuthology.lock.ops:unlocked: smithi031.front.sepia.ceph.com
2023-05-05T14:39:40.952 INFO:teuthology.lock.ops:unlocked: smithi120.front.sepia.ceph.com
2023-05-05T14:39:40.990 INFO:teuthology.lock.ops:unlocked: smithi136.front.sepia.ceph.com
2023-05-05T14:39:41.012 INFO:teuthology.lock.ops:unlocked: smithi190.front.sepia.ceph.com
2023-05-05T14:39:41.583 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/dispatcher/supervisor.py", line 56, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/dispatcher/supervisor.py", line 226, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/lock/ops.py", line 324, in reimage_machines
    log.info("Node '%s' reimaging is complete", machine)
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/provision/__init__.py", line 39, in reimage
    return obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/provision/fog.py", line 86, in create
    self.wait_for_deploy_task(task_id)
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/provision/fog.py", line 255, in wait_for_deploy_task
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/contextutil.py", line 135, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: reached maximum tries (60) after waiting for 900 seconds