2021-05-18T13:37:26.240 INFO:root:teuthology version: 1.1.0-19220a3b
2021-05-18T13:37:26.423 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba003.front.sepia.ceph.com' reimaging
2021-05-18T13:37:26.424 INFO:teuthology.lock.ops:Updating [ubuntu@gibba003.front.sepia.ceph.com]: reset os type and version on server
2021-05-18T13:37:26.424 INFO:teuthology.lock.ops:Updating gibba003.front.sepia.ceph.com on lock server
2021-05-18T13:37:26.439 INFO:teuthology.lock.ops:Node 'ubuntu@gibba003.front.sepia.ceph.com' reimaging is complete
2021-05-18T13:37:26.440 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba007.front.sepia.ceph.com' reimaging
2021-05-18T13:37:26.440 INFO:teuthology.lock.ops:Updating [ubuntu@gibba007.front.sepia.ceph.com]: reset os type and version on server
2021-05-18T13:37:26.440 INFO:teuthology.lock.ops:Updating gibba007.front.sepia.ceph.com on lock server
2021-05-18T13:37:26.455 INFO:teuthology.lock.ops:Node 'ubuntu@gibba007.front.sepia.ceph.com' reimaging is complete
2021-05-18T13:37:26.455 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba008.front.sepia.ceph.com' reimaging
2021-05-18T13:37:26.456 INFO:teuthology.lock.ops:Updating [ubuntu@gibba008.front.sepia.ceph.com]: reset os type and version on server
2021-05-18T13:37:26.456 INFO:teuthology.lock.ops:Updating gibba008.front.sepia.ceph.com on lock server
2021-05-18T13:37:26.470 INFO:teuthology.lock.ops:Node 'ubuntu@gibba008.front.sepia.ceph.com' reimaging is complete
2021-05-18T13:37:26.470 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba031.front.sepia.ceph.com' reimaging
2021-05-18T13:37:26.470 INFO:teuthology.lock.ops:Updating [ubuntu@gibba031.front.sepia.ceph.com]: reset os type and version on server
2021-05-18T13:37:26.471 INFO:teuthology.lock.ops:Updating gibba031.front.sepia.ceph.com on lock server
2021-05-18T13:37:26.485 INFO:teuthology.lock.ops:Node 'ubuntu@gibba031.front.sepia.ceph.com' reimaging is complete
2021-05-18T13:37:26.486 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba036.front.sepia.ceph.com' reimaging
2021-05-18T13:37:26.486 INFO:teuthology.lock.ops:Updating [ubuntu@gibba036.front.sepia.ceph.com]: reset os type and version on server
2021-05-18T13:37:26.486 INFO:teuthology.lock.ops:Updating gibba036.front.sepia.ceph.com on lock server
2021-05-18T13:37:26.500 INFO:teuthology.lock.ops:Node 'ubuntu@gibba036.front.sepia.ceph.com' reimaging is complete
2021-05-18T13:37:26.500 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba044.front.sepia.ceph.com' reimaging
2021-05-18T13:37:26.500 INFO:teuthology.lock.ops:Updating [ubuntu@gibba044.front.sepia.ceph.com]: reset os type and version on server
2021-05-18T13:37:26.501 INFO:teuthology.lock.ops:Updating gibba044.front.sepia.ceph.com on lock server
2021-05-18T13:37:26.515 INFO:teuthology.lock.ops:Node 'ubuntu@gibba044.front.sepia.ceph.com' reimaging is complete
2021-05-18T13:37:26.686 INFO:teuthology.provision.fog.gibba003:Scheduling deploy of centos 8.3
2021-05-18T13:37:26.690 INFO:teuthology.provision.fog.gibba007:Scheduling deploy of centos 8.3
2021-05-18T13:37:26.710 INFO:teuthology.provision.fog.gibba008:Scheduling deploy of centos 8.3
2021-05-18T13:37:26.729 INFO:teuthology.provision.fog.gibba031:Scheduling deploy of centos 8.3
2021-05-18T13:37:26.747 INFO:teuthology.provision.fog.gibba044:Scheduling deploy of centos 8.3
2021-05-18T13:37:26.765 INFO:teuthology.provision.fog.gibba036:Scheduling deploy of centos 8.3
2021-05-18T13:37:27.090 INFO:teuthology.orchestra.console:Power off gibba007
2021-05-18T13:37:27.132 INFO:teuthology.orchestra.console:Power off gibba003
2021-05-18T13:37:27.177 INFO:teuthology.orchestra.console:Power off gibba031
2021-05-18T13:37:27.193 INFO:teuthology.orchestra.console:Power off gibba008
2021-05-18T13:37:27.240 INFO:teuthology.orchestra.console:Power off gibba044
2021-05-18T13:37:27.267 INFO:teuthology.orchestra.console:Power off gibba036
2021-05-18T13:37:27.302 INFO:teuthology.orchestra.console:Power off for gibba003 completed
2021-05-18T13:37:27.404 INFO:teuthology.orchestra.console:Power on gibba003
2021-05-18T13:37:27.414 INFO:teuthology.orchestra.console:Power off for gibba036 completed
2021-05-18T13:37:27.516 INFO:teuthology.orchestra.console:Power on gibba036
2021-05-18T13:37:31.662 INFO:teuthology.orchestra.console:Power on for gibba003 completed
2021-05-18T13:37:31.763 INFO:teuthology.provision.fog.gibba003:Waiting for deploy to finish
2021-05-18T13:37:31.853 INFO:teuthology.orchestra.console:Power on for gibba036 completed
2021-05-18T13:37:31.954 INFO:teuthology.provision.fog.gibba036:Waiting for deploy to finish
2021-05-18T13:37:35.481 INFO:teuthology.orchestra.console:Power off for gibba007 completed
2021-05-18T13:37:35.575 INFO:teuthology.orchestra.console:Power off for gibba008 completed
2021-05-18T13:37:35.580 INFO:teuthology.orchestra.console:Power off for gibba031 completed
2021-05-18T13:37:35.583 INFO:teuthology.orchestra.console:Power on gibba007
2021-05-18T13:37:35.623 INFO:teuthology.orchestra.console:Power off for gibba044 completed
2021-05-18T13:37:35.676 INFO:teuthology.orchestra.console:Power on gibba008
2021-05-18T13:37:35.684 INFO:teuthology.orchestra.console:Power on gibba031
2021-05-18T13:37:35.724 INFO:teuthology.orchestra.console:Power on gibba044
2021-05-18T13:37:43.962 INFO:teuthology.orchestra.console:Power on for gibba007 completed
2021-05-18T13:37:44.054 INFO:teuthology.orchestra.console:Power on for gibba008 completed
2021-05-18T13:37:44.060 INFO:teuthology.orchestra.console:Power on for gibba031 completed
2021-05-18T13:37:44.064 INFO:teuthology.provision.fog.gibba007:Waiting for deploy to finish
2021-05-18T13:37:44.095 INFO:teuthology.orchestra.console:Power on for gibba044 completed
2021-05-18T13:37:44.155 INFO:teuthology.provision.fog.gibba008:Waiting for deploy to finish
2021-05-18T13:37:44.161 INFO:teuthology.provision.fog.gibba031:Waiting for deploy to finish
2021-05-18T13:37:44.197 INFO:teuthology.provision.fog.gibba044:Waiting for deploy to finish
2021-05-18T13:41:44.502 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-05-18T13:41:44.512 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-05-18T13:41:54.710 INFO:teuthology.provision.fog.gibba008:Node is ready
2021-05-18T13:41:54.723 INFO:teuthology.orchestra.run.gibba008.stdout:gibba029
2021-05-18T13:41:54.774 INFO:teuthology.orchestra.run.gibba008.stdout:172.21.2.129 gibba029.front.sepia.ceph.com gibba029
2021-05-18T13:41:55.064 INFO:teuthology.provision.fog.gibba008:Deploy complete!
2021-05-18T13:41:59.742 INFO:teuthology.provision.fog.gibba031:Node is ready
2021-05-18T13:41:59.754 INFO:teuthology.orchestra.run.gibba031.stdout:gibba031.front.sepia.ceph.com
2021-05-18T13:41:59.807 INFO:teuthology.orchestra.run.gibba031.stdout:172.21.2.131 gibba031.front.sepia.ceph.com gibba031
2021-05-18T13:42:00.097 INFO:teuthology.provision.fog.gibba031:Deploy complete!
2021-05-18T13:42:06.548 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-05-18T13:42:06.823 INFO:teuthology.provision.fog.gibba003:Node is ready
2021-05-18T13:42:06.834 INFO:teuthology.orchestra.run.gibba003.stdout:gibba003.front.sepia.ceph.com
2021-05-18T13:42:06.887 INFO:teuthology.orchestra.run.gibba003.stdout:172.21.2.103 gibba003.front.sepia.ceph.com gibba003
2021-05-18T13:42:07.176 INFO:teuthology.provision.fog.gibba003:Deploy complete!
2021-05-18T13:42:09.872 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-05-18T13:42:09.912 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-05-18T13:42:10.138 INFO:teuthology.provision.fog.gibba007:Node is ready
2021-05-18T13:42:10.190 INFO:teuthology.orchestra.run.gibba007.stdout:gibba007.front.sepia.ceph.com
2021-05-18T13:42:10.202 INFO:teuthology.provision.fog.gibba044:Node is ready
2021-05-18T13:42:10.242 INFO:teuthology.orchestra.run.gibba007.stdout:172.21.2.107 gibba007.front.sepia.ceph.com gibba007
2021-05-18T13:42:10.253 INFO:teuthology.orchestra.run.gibba044.stdout:gibba044.front.sepia.ceph.com
2021-05-18T13:42:10.305 INFO:teuthology.orchestra.run.gibba044.stdout:172.21.2.144 gibba044.front.sepia.ceph.com gibba044
2021-05-18T13:42:10.531 INFO:teuthology.provision.fog.gibba007:Deploy complete!
2021-05-18T13:42:10.609 INFO:teuthology.provision.fog.gibba044:Deploy complete!
2021-05-18T13:52:21.714 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/dispatcher/supervisor.py", line 163, in reimage
    reimaged = reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/lock/ops.py", line 317, in reimage_machines
    log.info("Node '%s' reimaging is complete", machine)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/provision/__init__.py", line 39, in reimage
    return obj.create()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/provision/fog.py", line 86, in create
    self.wait_for_deploy_task(task_id)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/provision/fog.py", line 244, in wait_for_deploy_task
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/contextutil.py", line 133, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: reached maximum tries (60) after waiting for 900 seconds
2021-05-18T13:52:21.759 INFO:teuthology.nuke:Checking targets against current locks
2021-05-18T13:52:21.856 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-18T13:52:21.863 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-18T13:52:21.864 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-18T13:52:21.865 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-18T13:52:21.866 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-18T13:52:21.868 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-18T13:52:21.925 INFO:teuthology.orchestra.console:Power off gibba044
2021-05-18T13:52:21.944 INFO:teuthology.orchestra.console:Power off gibba036
2021-05-18T13:52:21.955 INFO:teuthology.orchestra.console:Power off gibba003
2021-05-18T13:52:21.969 INFO:teuthology.orchestra.console:Power off gibba007
2021-05-18T13:52:21.992 INFO:teuthology.orchestra.console:Power off gibba031
2021-05-18T13:52:22.026 INFO:teuthology.orchestra.console:Power off gibba008
2021-05-18T13:52:30.361 INFO:teuthology.orchestra.console:Power off for gibba007 completed
2021-05-18T13:52:30.368 INFO:teuthology.orchestra.console:Power off for gibba044 completed
2021-05-18T13:52:30.373 INFO:teuthology.orchestra.console:Power off for gibba003 completed
2021-05-18T13:52:30.395 INFO:teuthology.orchestra.console:Power off for gibba008 completed
2021-05-18T13:52:30.443 INFO:teuthology.orchestra.console:Power off for gibba031 completed
2021-05-18T13:52:30.507 INFO:teuthology.lock.ops:unlocked gibba044.front.sepia.ceph.com
2021-05-18T13:52:30.510 INFO:teuthology.lock.ops:unlocked gibba007.front.sepia.ceph.com
2021-05-18T13:52:30.518 ERROR:teuthology.lock.ops:failed to unlock gibba003.front.sepia.ceph.com. reason: 500
2021-05-18T13:52:30.543 INFO:teuthology.lock.ops:unlocked gibba008.front.sepia.ceph.com
2021-05-18T13:52:30.579 INFO:teuthology.lock.ops:unlocked gibba031.front.sepia.ceph.com
2021-05-18T13:52:34.460 INFO:teuthology.orchestra.console:Power off for gibba036 completed
2021-05-18T13:52:34.598 INFO:teuthology.lock.ops:unlocked gibba036.front.sepia.ceph.com
2021-05-18T13:52:34.599 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/virtualenv/bin/teuthology-dispatcher", line 33, in <module>
    sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-dispatcher')())
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/scripts/dispatcher.py", line 33, in main
    teuthology.dispatcher.main(args)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/dispatcher/__init__.py", line 61, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/dispatcher/supervisor.py", line 50, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/dispatcher/supervisor.py", line 163, in reimage
    reimaged = reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/lock/ops.py", line 317, in reimage_machines
    log.info("Node '%s' reimaging is complete", machine)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/provision/__init__.py", line 39, in reimage
    return obj.create()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/provision/fog.py", line 86, in create
    self.wait_for_deploy_task(task_id)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/provision/fog.py", line 244, in wait_for_deploy_task
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/contextutil.py", line 133, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: reached maximum tries (60) after waiting for 900 seconds