2021-06-19T14:43:59.331 INFO:root:teuthology version: 1.1.0-f359b10d 2021-06-19T14:43:59.623 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba009.front.sepia.ceph.com' reimaging 2021-06-19T14:43:59.624 INFO:teuthology.lock.ops:Updating [ubuntu@gibba009.front.sepia.ceph.com]: reset os type and version on server 2021-06-19T14:43:59.624 INFO:teuthology.lock.ops:Updating gibba009.front.sepia.ceph.com on lock server 2021-06-19T14:43:59.639 INFO:teuthology.lock.ops:Node 'ubuntu@gibba009.front.sepia.ceph.com' reimaging is complete 2021-06-19T14:43:59.640 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba010.front.sepia.ceph.com' reimaging 2021-06-19T14:43:59.640 INFO:teuthology.lock.ops:Updating [ubuntu@gibba010.front.sepia.ceph.com]: reset os type and version on server 2021-06-19T14:43:59.640 INFO:teuthology.lock.ops:Updating gibba010.front.sepia.ceph.com on lock server 2021-06-19T14:43:59.655 INFO:teuthology.lock.ops:Node 'ubuntu@gibba010.front.sepia.ceph.com' reimaging is complete 2021-06-19T14:43:59.656 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba016.front.sepia.ceph.com' reimaging 2021-06-19T14:43:59.656 INFO:teuthology.lock.ops:Updating [ubuntu@gibba016.front.sepia.ceph.com]: reset os type and version on server 2021-06-19T14:43:59.656 INFO:teuthology.lock.ops:Updating gibba016.front.sepia.ceph.com on lock server 2021-06-19T14:43:59.670 INFO:teuthology.lock.ops:Node 'ubuntu@gibba016.front.sepia.ceph.com' reimaging is complete 2021-06-19T14:43:59.671 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba026.front.sepia.ceph.com' reimaging 2021-06-19T14:43:59.671 INFO:teuthology.lock.ops:Updating [ubuntu@gibba026.front.sepia.ceph.com]: reset os type and version on server 2021-06-19T14:43:59.671 INFO:teuthology.lock.ops:Updating gibba026.front.sepia.ceph.com on lock server 2021-06-19T14:43:59.686 INFO:teuthology.lock.ops:Node 'ubuntu@gibba026.front.sepia.ceph.com' reimaging is complete 2021-06-19T14:43:59.686 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba039.front.sepia.ceph.com' reimaging 2021-06-19T14:43:59.687 INFO:teuthology.lock.ops:Updating [ubuntu@gibba039.front.sepia.ceph.com]: reset os type and version on server 2021-06-19T14:43:59.687 INFO:teuthology.lock.ops:Updating gibba039.front.sepia.ceph.com on lock server 2021-06-19T14:43:59.706 INFO:teuthology.lock.ops:Node 'ubuntu@gibba039.front.sepia.ceph.com' reimaging is complete 2021-06-19T14:43:59.707 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba044.front.sepia.ceph.com' reimaging 2021-06-19T14:43:59.707 INFO:teuthology.lock.ops:Updating [ubuntu@gibba044.front.sepia.ceph.com]: reset os type and version on server 2021-06-19T14:43:59.707 INFO:teuthology.lock.ops:Updating gibba044.front.sepia.ceph.com on lock server 2021-06-19T14:43:59.721 INFO:teuthology.lock.ops:Node 'ubuntu@gibba044.front.sepia.ceph.com' reimaging is complete 2021-06-19T14:43:59.864 INFO:teuthology.provision.fog.gibba009:Scheduling deploy of rhel 7.9 2021-06-19T14:43:59.895 INFO:teuthology.provision.fog.gibba016:Scheduling deploy of rhel 7.9 2021-06-19T14:43:59.932 INFO:teuthology.provision.fog.gibba010:Scheduling deploy of rhel 7.9 2021-06-19T14:43:59.939 INFO:teuthology.provision.fog.gibba026:Scheduling deploy of rhel 7.9 2021-06-19T14:43:59.960 INFO:teuthology.provision.fog.gibba039:Scheduling deploy of rhel 7.9 2021-06-19T14:43:59.963 INFO:teuthology.provision.fog.gibba044:Scheduling deploy of rhel 7.9 2021-06-19T14:44:00.263 INFO:teuthology.orchestra.console:Power off gibba009 2021-06-19T14:44:00.306 INFO:teuthology.orchestra.console:Power off gibba016 2021-06-19T14:44:00.333 INFO:teuthology.orchestra.console:Power off gibba010 2021-06-19T14:44:00.346 INFO:teuthology.orchestra.console:Power off gibba044 2021-06-19T14:44:00.364 INFO:teuthology.orchestra.console:Power off gibba039 2021-06-19T14:44:00.406 INFO:teuthology.orchestra.console:Power off gibba026 2021-06-19T14:44:00.421 INFO:teuthology.orchestra.console:Power off for gibba009 completed 2021-06-19T14:44:00.489 INFO:teuthology.orchestra.console:Power off for gibba016 completed 2021-06-19T14:44:00.499 INFO:teuthology.orchestra.console:Power off for gibba044 completed 2021-06-19T14:44:00.516 INFO:teuthology.orchestra.console:Power off for gibba039 completed 2021-06-19T14:44:00.516 INFO:teuthology.orchestra.console:Power off for gibba010 completed 2021-06-19T14:44:00.523 INFO:teuthology.orchestra.console:Power on gibba009 2021-06-19T14:44:00.558 INFO:teuthology.orchestra.console:Power off for gibba026 completed 2021-06-19T14:44:00.591 INFO:teuthology.orchestra.console:Power on gibba016 2021-06-19T14:44:00.600 INFO:teuthology.orchestra.console:Power on gibba044 2021-06-19T14:44:00.622 INFO:teuthology.orchestra.console:Power on gibba039 2021-06-19T14:44:00.629 INFO:teuthology.orchestra.console:Power on gibba010 2021-06-19T14:44:00.660 INFO:teuthology.orchestra.console:Power on gibba026 2021-06-19T14:44:04.998 INFO:teuthology.orchestra.console:Power on for gibba026 completed 2021-06-19T14:44:05.099 INFO:teuthology.provision.fog.gibba026:Waiting for deploy to finish 2021-06-19T14:44:08.903 INFO:teuthology.orchestra.console:Power on for gibba009 completed 2021-06-19T14:44:08.963 INFO:teuthology.orchestra.console:Power on for gibba016 completed 2021-06-19T14:44:08.987 INFO:teuthology.orchestra.console:Power on for gibba044 completed 2021-06-19T14:44:09.004 INFO:teuthology.orchestra.console:Power on for gibba010 completed 2021-06-19T14:44:09.005 INFO:teuthology.provision.fog.gibba009:Waiting for deploy to finish 2021-06-19T14:44:09.064 INFO:teuthology.provision.fog.gibba016:Waiting for deploy to finish 2021-06-19T14:44:09.088 INFO:teuthology.provision.fog.gibba044:Waiting for deploy to finish 2021-06-19T14:44:09.105 INFO:teuthology.provision.fog.gibba010:Waiting for deploy to finish 2021-06-19T14:44:09.197 INFO:teuthology.orchestra.console:Power on for gibba039 completed 2021-06-19T14:44:09.298 INFO:teuthology.provision.fog.gibba039:Waiting for deploy to finish 2021-06-19T14:48:32.569 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-19T14:48:43.612 INFO:teuthology.provision.fog.gibba039:Node is ready 2021-06-19T14:48:43.960 INFO:teuthology.orchestra.run.gibba039.stdout:gibba018 2021-06-19T14:48:44.296 INFO:teuthology.orchestra.run.gibba039.stdout:172.21.2.118 gibba018.front.sepia.ceph.com gibba018 2021-06-19T14:48:45.568 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-19T14:48:45.578 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-19T14:48:46.119 INFO:teuthology.provision.fog.gibba039:Deploy complete! 2021-06-19T14:48:46.198 INFO:teuthology.provision.fog.gibba044:Node is ready 2021-06-19T14:48:46.218 INFO:teuthology.provision.fog.gibba016:Node is ready 2021-06-19T14:48:46.540 INFO:teuthology.orchestra.run.gibba044.stdout:gibba044.front.sepia.ceph.com 2021-06-19T14:48:46.561 INFO:teuthology.orchestra.run.gibba016.stdout:gibba016.front.sepia.ceph.com 2021-06-19T14:48:46.874 INFO:teuthology.orchestra.run.gibba044.stdout:172.21.2.144 gibba044.front.sepia.ceph.com gibba044 2021-06-19T14:48:46.895 INFO:teuthology.orchestra.run.gibba016.stdout:172.21.2.116 gibba016.front.sepia.ceph.com gibba016 2021-06-19T14:48:48.063 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-19T14:48:48.612 INFO:teuthology.provision.fog.gibba044:Deploy complete! 2021-06-19T14:48:48.643 INFO:teuthology.provision.fog.gibba016:Deploy complete! 2021-06-19T14:48:48.733 INFO:teuthology.provision.fog.gibba026:Node is ready 2021-06-19T14:48:49.078 INFO:teuthology.orchestra.run.gibba026.stdout:gibba026.front.sepia.ceph.com 2021-06-19T14:48:49.413 INFO:teuthology.orchestra.run.gibba026.stdout:172.21.2.126 gibba026.front.sepia.ceph.com gibba026 2021-06-19T14:48:51.148 INFO:teuthology.provision.fog.gibba026:Deploy complete! 2021-06-19T14:48:52.145 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-19T14:48:52.766 INFO:teuthology.provision.fog.gibba010:Node is ready 2021-06-19T14:48:53.106 INFO:teuthology.orchestra.run.gibba010.stdout:gibba010.front.sepia.ceph.com 2021-06-19T14:48:53.441 INFO:teuthology.orchestra.run.gibba010.stdout:172.21.2.110 gibba010.front.sepia.ceph.com gibba010 2021-06-19T14:48:55.193 INFO:teuthology.provision.fog.gibba010:Deploy complete! 2021-06-19T14:59:01.345 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 163, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/lock/ops.py", line 317, in reimage_machines log.info("Node '%s' reimaging is complete", machine) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/provision/__init__.py", line 39, in reimage return obj.create() File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/provision/fog.py", line 86, in create self.wait_for_deploy_task(task_id) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/provision/fog.py", line 244, in wait_for_deploy_task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (60) after waiting for 900 seconds 2021-06-19T14:59:01.394 INFO:teuthology.nuke:Checking targets against current locks 2021-06-19T14:59:01.518 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-19T14:59:01.523 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-19T14:59:01.524 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-19T14:59:01.526 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-19T14:59:01.531 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-19T14:59:01.532 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-19T14:59:01.578 INFO:teuthology.orchestra.console:Power off gibba016 2021-06-19T14:59:01.598 INFO:teuthology.orchestra.console:Power off gibba010 2021-06-19T14:59:01.612 INFO:teuthology.orchestra.console:Power off gibba009 2021-06-19T14:59:01.672 INFO:teuthology.orchestra.console:Power off gibba044 2021-06-19T14:59:01.685 INFO:teuthology.orchestra.console:Power off gibba039 2021-06-19T14:59:01.701 INFO:teuthology.orchestra.console:Power off gibba026 2021-06-19T14:59:10.052 INFO:teuthology.orchestra.console:Power off for gibba016 completed 2021-06-19T14:59:10.060 INFO:teuthology.orchestra.console:Power off for gibba010 completed 2021-06-19T14:59:10.069 INFO:teuthology.orchestra.console:Power off for gibba039 completed 2021-06-19T14:59:10.075 INFO:teuthology.orchestra.console:Power off for gibba044 completed 2021-06-19T14:59:10.083 INFO:teuthology.orchestra.console:Power off for gibba026 completed 2021-06-19T14:59:10.215 INFO:teuthology.lock.ops:unlocked gibba016.front.sepia.ceph.com 2021-06-19T14:59:10.219 ERROR:teuthology.lock.ops:failed to unlock gibba010.front.sepia.ceph.com. reason: 500 2021-06-19T14:59:10.232 INFO:teuthology.lock.ops:unlocked gibba026.front.sepia.ceph.com 2021-06-19T14:59:10.233 ERROR:teuthology.lock.ops:failed to unlock gibba039.front.sepia.ceph.com. reason: 500 2021-06-19T14:59:10.239 INFO:teuthology.lock.ops:unlocked gibba044.front.sepia.ceph.com 2021-06-19T14:59:14.156 INFO:teuthology.orchestra.console:Power off for gibba009 completed 2021-06-19T14:59:14.298 INFO:teuthology.lock.ops:unlocked gibba009.front.sepia.ceph.com 2021-06-19T14:59:14.299 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/bin/teuthology-dispatcher", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-dispatcher')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/scripts/dispatcher.py", line 33, in main teuthology.dispatcher.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/__init__.py", line 61, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 50, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 163, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/lock/ops.py", line 317, in reimage_machines log.info("Node '%s' reimaging is complete", machine) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/provision/__init__.py", line 39, in reimage return obj.create() File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/provision/fog.py", line 86, in create self.wait_for_deploy_task(task_id) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/provision/fog.py", line 244, in wait_for_deploy_task while proceed(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (60) after waiting for 900 seconds