2021-04-01T18:59:13.353 INFO:root:teuthology version: 1.1.0-85d61eae 2021-04-01T18:59:13.518 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba021.front.sepia.ceph.com' reimaging 2021-04-01T18:59:13.519 INFO:teuthology.lock.ops:Updating [ubuntu@gibba021.front.sepia.ceph.com]: reset os type and version on server 2021-04-01T18:59:13.520 INFO:teuthology.lock.ops:Updating gibba021.front.sepia.ceph.com on lock server 2021-04-01T18:59:13.541 INFO:teuthology.lock.ops:Node 'ubuntu@gibba021.front.sepia.ceph.com' reimaging is complete 2021-04-01T18:59:13.541 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba022.front.sepia.ceph.com' reimaging 2021-04-01T18:59:13.541 INFO:teuthology.lock.ops:Updating [ubuntu@gibba022.front.sepia.ceph.com]: reset os type and version on server 2021-04-01T18:59:13.542 INFO:teuthology.lock.ops:Updating gibba022.front.sepia.ceph.com on lock server 2021-04-01T18:59:13.561 INFO:teuthology.lock.ops:Node 'ubuntu@gibba022.front.sepia.ceph.com' reimaging is complete 2021-04-01T18:59:13.562 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba030.front.sepia.ceph.com' reimaging 2021-04-01T18:59:13.562 INFO:teuthology.lock.ops:Updating [ubuntu@gibba030.front.sepia.ceph.com]: reset os type and version on server 2021-04-01T18:59:13.563 INFO:teuthology.lock.ops:Updating gibba030.front.sepia.ceph.com on lock server 2021-04-01T18:59:13.581 INFO:teuthology.lock.ops:Node 'ubuntu@gibba030.front.sepia.ceph.com' reimaging is complete 2021-04-01T18:59:13.582 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba038.front.sepia.ceph.com' reimaging 2021-04-01T18:59:13.582 INFO:teuthology.lock.ops:Updating [ubuntu@gibba038.front.sepia.ceph.com]: reset os type and version on server 2021-04-01T18:59:13.582 INFO:teuthology.lock.ops:Updating gibba038.front.sepia.ceph.com on lock server 2021-04-01T18:59:13.598 INFO:teuthology.lock.ops:Node 'ubuntu@gibba038.front.sepia.ceph.com' reimaging is complete 2021-04-01T18:59:13.778 INFO:teuthology.provision.fog.gibba021:Scheduling deploy of rhel 7.9 2021-04-01T18:59:13.789 INFO:teuthology.provision.fog.gibba022:Scheduling deploy of rhel 7.9 2021-04-01T18:59:13.807 INFO:teuthology.provision.fog.gibba030:Scheduling deploy of rhel 7.9 2021-04-01T18:59:13.821 INFO:teuthology.provision.fog.gibba038:Scheduling deploy of rhel 7.9 2021-04-01T18:59:14.113 INFO:teuthology.orchestra.console:Power off gibba022 2021-04-01T18:59:14.131 INFO:teuthology.orchestra.console:Power off gibba030 2021-04-01T18:59:14.177 INFO:teuthology.orchestra.console:Power off gibba038 2021-04-01T18:59:14.191 INFO:teuthology.orchestra.console:Power off gibba021 2021-04-01T18:59:14.286 INFO:teuthology.orchestra.console:Power off for gibba022 completed 2021-04-01T18:59:14.299 INFO:teuthology.orchestra.console:Power off for gibba030 completed 2021-04-01T18:59:14.327 INFO:teuthology.orchestra.console:Power off for gibba038 completed 2021-04-01T18:59:14.338 INFO:teuthology.orchestra.console:Power off for gibba021 completed 2021-04-01T18:59:14.389 INFO:teuthology.orchestra.console:Power on gibba022 2021-04-01T18:59:14.400 INFO:teuthology.orchestra.console:Power on gibba030 2021-04-01T18:59:14.429 INFO:teuthology.orchestra.console:Power on gibba038 2021-04-01T18:59:14.438 INFO:teuthology.orchestra.console:Power on gibba021 2021-04-01T18:59:18.702 INFO:teuthology.orchestra.console:Power on for gibba038 completed 2021-04-01T18:59:18.712 INFO:teuthology.orchestra.console:Power on for gibba021 completed 2021-04-01T18:59:18.805 INFO:teuthology.provision.fog.gibba038:Waiting for deploy to finish 2021-04-01T18:59:18.813 INFO:teuthology.provision.fog.gibba021:Waiting for deploy to finish 2021-04-01T18:59:22.792 INFO:teuthology.orchestra.console:Power on for gibba022 completed 2021-04-01T18:59:22.806 INFO:teuthology.orchestra.console:Power on for gibba030 completed 2021-04-01T18:59:22.894 INFO:teuthology.provision.fog.gibba022:Waiting for deploy to finish 2021-04-01T18:59:22.907 INFO:teuthology.provision.fog.gibba030:Waiting for deploy to finish 2021-04-01T19:03:22.919 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-01T19:03:27.714 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-01T19:03:38.715 INFO:teuthology.provision.fog.gibba022:Node is ready 2021-04-01T19:03:39.065 INFO:teuthology.orchestra.run.gibba022.stdout:gibba022.front.sepia.ceph.com 2021-04-01T19:03:39.405 INFO:teuthology.orchestra.run.gibba022.stdout:172.21.2.122 gibba022.front.sepia.ceph.com gibba022 2021-04-01T19:03:41.206 INFO:teuthology.provision.fog.gibba022:Deploy complete! 2021-04-01T19:03:43.668 INFO:teuthology.provision.fog.gibba021:Node is ready 2021-04-01T19:03:44.021 INFO:teuthology.orchestra.run.gibba021.stdout:gibba021.front.sepia.ceph.com 2021-04-01T19:03:44.361 INFO:teuthology.orchestra.run.gibba021.stdout:172.21.2.121 gibba021.front.sepia.ceph.com gibba021 2021-04-01T19:03:46.139 INFO:teuthology.provision.fog.gibba021:Deploy complete! 2021-04-01T19:03:48.531 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-01T19:03:49.175 INFO:teuthology.provision.fog.gibba030:Node is ready 2021-04-01T19:03:49.521 INFO:teuthology.orchestra.run.gibba030.stdout:gibba030.front.sepia.ceph.com 2021-04-01T19:03:49.857 INFO:teuthology.orchestra.run.gibba030.stdout:172.21.2.130 gibba030.front.sepia.ceph.com gibba030 2021-04-01T19:03:51.631 INFO:teuthology.provision.fog.gibba030:Deploy complete! 2021-04-01T19:03:53.452 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-01T19:03:54.075 INFO:teuthology.provision.fog.gibba038:Node is ready 2021-04-01T19:03:54.413 INFO:teuthology.orchestra.run.gibba038.stdout:gibba038.front.sepia.ceph.com 2021-04-01T19:03:54.748 INFO:teuthology.orchestra.run.gibba038.stdout:172.21.2.138 gibba038.front.sepia.ceph.com gibba038 2021-04-01T19:03:56.529 INFO:teuthology.provision.fog.gibba038:Deploy complete! 2021-04-01T19:03:56.699 INFO:teuthology.lock.ops:Checking gibba021.front.sepia.ceph.com 2021-04-01T19:03:56.700 INFO:teuthology.lock.ops:Checking gibba022.front.sepia.ceph.com 2021-04-01T19:03:56.701 INFO:teuthology.lock.ops:New key found. Updating... 2021-04-01T19:03:56.719 INFO:teuthology.lock.ops:Checking gibba030.front.sepia.ceph.com 2021-04-01T19:03:56.719 INFO:teuthology.lock.ops:New key found. Updating... 2021-04-01T19:03:56.733 INFO:teuthology.lock.ops:Checking gibba038.front.sepia.ceph.com 2021-04-01T19:03:56.733 INFO:teuthology.lock.ops:New key found. Updating... 2021-04-01T19:03:56.749 INFO:teuthology.lock.ops:Updating [gibba021.front.sepia.ceph.com]: set os type and version on server 2021-04-01T19:03:56.749 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-01T19:03:57.647 INFO:teuthology.orchestra.run.gibba021.stdout:x86_64 2021-04-01T19:03:57.750 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/dispatcher/supervisor.py", line 163, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/lock/ops.py", line 319, in reimage_machines update_nodes(reimaged) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/lock/ops.py", line 38, in update_nodes inventory_info = remote.inventory_info File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/remote.py", line 644, in inventory_info node['os_type'] = self.os.name File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/remote.py", line 614, in os os_release = self.sh('cat /etc/os-release').strip() File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/remote.py", line 267, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/run.py", line 175, in _raise_for_status node=self.hostname) teuthology.exceptions.ConnectionLostError: SSH connection to gibba021 was lost: 'cat /etc/os-release' 2021-04-01T19:03:57.800 INFO:teuthology.nuke:Checking targets against current locks 2021-04-01T19:03:57.888 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-01T19:03:57.901 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-01T19:03:57.913 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-01T19:03:57.925 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-01T19:03:58.043 INFO:teuthology.orchestra.console:Power off gibba030 2021-04-01T19:03:58.068 INFO:teuthology.orchestra.console:Power off gibba022 2021-04-01T19:03:58.083 INFO:teuthology.orchestra.console:Power off gibba021 2021-04-01T19:03:58.127 INFO:teuthology.orchestra.console:Power off gibba038 2021-04-01T19:04:02.399 INFO:teuthology.orchestra.console:Power off for gibba021 completed 2021-04-01T19:04:02.557 INFO:teuthology.lock.ops:unlocked gibba021.front.sepia.ceph.com 2021-04-01T19:04:06.477 INFO:teuthology.orchestra.console:Power off for gibba030 completed 2021-04-01T19:04:06.503 INFO:teuthology.orchestra.console:Power off for gibba022 completed 2021-04-01T19:04:06.591 INFO:teuthology.orchestra.console:Power off for gibba038 completed 2021-04-01T19:04:06.632 INFO:teuthology.lock.ops:unlocked gibba030.front.sepia.ceph.com 2021-04-01T19:04:06.689 INFO:teuthology.lock.ops:unlocked gibba022.front.sepia.ceph.com 2021-04-01T19:04:06.730 INFO:teuthology.lock.ops:unlocked gibba038.front.sepia.ceph.com 2021-04-01T19:04:06.730 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/bin/teuthology-dispatcher", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-dispatcher')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/scripts/dispatcher.py", line 33, in main teuthology.dispatcher.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/dispatcher/__init__.py", line 61, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/dispatcher/supervisor.py", line 50, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/dispatcher/supervisor.py", line 163, in reimage reimaged = reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/lock/ops.py", line 319, in reimage_machines update_nodes(reimaged) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/lock/ops.py", line 38, in update_nodes inventory_info = remote.inventory_info File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/remote.py", line 644, in inventory_info node['os_type'] = self.os.name File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/remote.py", line 614, in os os_release = self.sh('cat /etc/os-release').strip() File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/remote.py", line 267, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/run.py", line 175, in _raise_for_status node=self.hostname) teuthology.exceptions.ConnectionLostError: SSH connection to gibba021 was lost: 'cat /etc/os-release'