2024-04-18T05:22:43.915 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d
2024-04-18T05:22:43.915 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-04-18T05:22:44.081 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7fb0b8563f70>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7fb0b95c3c10>, signals=[15])
2024-04-18T05:22:44.082 INFO:teuthology.lock.ops:Start node 'smithi077.front.sepia.ceph.com' reimaging
2024-04-18T05:22:44.082 INFO:teuthology.lock.ops:Updating [smithi077.front.sepia.ceph.com]: reset os type and version on server
2024-04-18T05:22:44.082 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server
2024-04-18T05:22:44.101 INFO:teuthology.lock.ops:Start node 'smithi100.front.sepia.ceph.com' reimaging
2024-04-18T05:22:44.101 INFO:teuthology.lock.ops:Updating [smithi100.front.sepia.ceph.com]: reset os type and version on server
2024-04-18T05:22:44.101 INFO:teuthology.lock.ops:Updating smithi100.front.sepia.ceph.com on lock server
2024-04-18T05:22:44.117 INFO:teuthology.lock.ops:Start node 'smithi144.front.sepia.ceph.com' reimaging
2024-04-18T05:22:44.117 INFO:teuthology.lock.ops:Updating [smithi144.front.sepia.ceph.com]: reset os type and version on server
2024-04-18T05:22:44.118 INFO:teuthology.lock.ops:Updating smithi144.front.sepia.ceph.com on lock server
2024-04-18T05:22:44.213 DEBUG:teuthology.provision.fog.smithi077:Requesting image smithi_rhel_8.6 (ID 5)
2024-04-18T05:22:44.233 DEBUG:teuthology.provision.fog.smithi100:Requesting image smithi_rhel_8.6 (ID 5)
2024-04-18T05:22:44.236 DEBUG:teuthology.provision.fog.smithi144:Requesting image smithi_rhel_8.6 (ID 5)
2024-04-18T05:22:44.254 INFO:teuthology.provision.fog.smithi077:Scheduling deploy of rhel 8.6
2024-04-18T05:22:44.276 INFO:teuthology.provision.fog.smithi100:Scheduling deploy of rhel 8.6
2024-04-18T05:22:44.279 INFO:teuthology.provision.fog.smithi144:Scheduling deploy of rhel 8.6
2024-04-18T05:22:44.596 INFO:teuthology.orchestra.console.smithi077:Power off
2024-04-18T05:22:44.596 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-18T05:22:44.610 INFO:teuthology.orchestra.console.smithi100:Power off
2024-04-18T05:22:44.610 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-18T05:22:44.622 INFO:teuthology.orchestra.console.smithi144:Power off
2024-04-18T05:22:44.622 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-18T05:22:44.630 DEBUG:teuthology.orchestra.console.smithi077:power off output: Chassis Power Control: Down/Off
2024-04-18T05:22:44.631 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T05:22:44.638 DEBUG:teuthology.orchestra.console.smithi100:power off output: Chassis Power Control: Down/Off
2024-04-18T05:22:44.639 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T05:22:44.647 DEBUG:teuthology.orchestra.console.smithi144:power off output: Chassis Power Control: Down/Off
2024-04-18T05:22:44.647 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T05:22:44.656 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off
2024-04-18T05:22:44.660 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is off
2024-04-18T05:22:44.669 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is off
2024-04-18T05:22:44.758 INFO:teuthology.orchestra.console.smithi077:Power off completed
2024-04-18T05:22:44.761 INFO:teuthology.orchestra.console.smithi100:Power off completed
2024-04-18T05:22:44.770 INFO:teuthology.orchestra.console.smithi144:Power off completed
2024-04-18T05:22:44.860 INFO:teuthology.orchestra.console.smithi077:Power on
2024-04-18T05:22:44.860 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-04-18T05:22:44.867 INFO:teuthology.orchestra.console.smithi100:Power on
2024-04-18T05:22:44.867 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-04-18T05:22:44.874 INFO:teuthology.orchestra.console.smithi144:Power on
2024-04-18T05:22:44.874 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-04-18T05:22:44.881 DEBUG:teuthology.orchestra.console.smithi077:power on output: Chassis Power Control: Up/On
2024-04-18T05:22:44.882 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T05:22:44.891 DEBUG:teuthology.orchestra.console.smithi100:power on output: Chassis Power Control: Up/On
2024-04-18T05:22:44.891 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T05:22:44.900 DEBUG:teuthology.orchestra.console.smithi144:power on output: Chassis Power Control: Up/On
2024-04-18T05:22:44.901 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T05:22:44.909 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off
2024-04-18T05:22:44.913 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is off
2024-04-18T05:22:44.921 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is off
2024-04-18T05:22:48.914 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T05:22:48.923 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T05:22:48.929 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T05:22:49.025 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off
2024-04-18T05:22:49.031 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is on
2024-04-18T05:22:49.037 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is off
2024-04-18T05:22:49.132 INFO:teuthology.orchestra.console.smithi100:Power on completed
2024-04-18T05:22:49.232 INFO:teuthology.provision.fog.smithi100:Waiting for deploy to finish
2024-04-18T05:22:53.030 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T05:22:53.038 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T05:22:53.139 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on
2024-04-18T05:22:53.146 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is on
2024-04-18T05:22:53.240 INFO:teuthology.orchestra.console.smithi077:Power on completed
2024-04-18T05:22:53.247 INFO:teuthology.orchestra.console.smithi144:Power on completed
2024-04-18T05:22:53.341 INFO:teuthology.provision.fog.smithi077:Waiting for deploy to finish
2024-04-18T05:22:53.349 INFO:teuthology.provision.fog.smithi144:Waiting for deploy to finish
2024-04-18T05:22:53.501 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:23:12.033 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144
2024-04-18T05:23:18.039 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:23:21.118 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144
2024-04-18T05:23:27.123 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:23:30.177 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144
2024-04-18T05:23:36.180 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:23:39.230 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144
2024-04-18T05:23:45.235 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:23:48.286 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144
2024-04-18T05:23:54.288 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:23:57.342 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144
2024-04-18T05:24:03.348 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:24:09.470 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144
2024-04-18T05:24:15.475 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:24:18.526 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144
2024-04-18T05:24:24.527 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:24:27.255 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144
2024-04-18T05:24:33.262 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:24:33.389 ERROR:teuthology.orchestra.connection:Error authenticating with smithi144.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-04-18T05:24:37.516 ERROR:teuthology.orchestra.connection:Error authenticating with smithi144.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-04-18T05:24:44.640 ERROR:teuthology.orchestra.connection:Error authenticating with smithi144.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-04-18T05:24:54.766 ERROR:teuthology.orchestra.connection:Error authenticating with smithi144.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-04-18T05:25:07.890 ERROR:teuthology.orchestra.connection:Error authenticating with smithi144.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-04-18T05:25:24.016 ERROR:teuthology.orchestra.connection:Error authenticating with smithi144.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-04-18T05:25:36.138 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:25:39.202 WARNING:teuthology.provision.fog.smithi100:[Errno None] Unable to connect to port 22 on 172.21.15.100
2024-04-18T05:25:43.019 ERROR:paramiko.transport:Exception (client): Error reading SSH protocol banner
2024-04-18T05:25:43.021 ERROR:paramiko.transport:Traceback (most recent call last):
2024-04-18T05:25:43.021 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2327, in _check_banner
2024-04-18T05:25:43.021 ERROR:paramiko.transport:    buf = self.packetizer.readline(timeout)
2024-04-18T05:25:43.021 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 381, in readline
2024-04-18T05:25:43.021 ERROR:paramiko.transport:    buf += self._read_timeout(timeout)
2024-04-18T05:25:43.021 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 618, in _read_timeout
2024-04-18T05:25:43.021 ERROR:paramiko.transport:    raise EOFError()
2024-04-18T05:25:43.021 ERROR:paramiko.transport:EOFError
2024-04-18T05:25:43.021 ERROR:paramiko.transport:
2024-04-18T05:25:43.021 ERROR:paramiko.transport:During handling of the above exception, another exception occurred:
2024-04-18T05:25:43.021 ERROR:paramiko.transport:
2024-04-18T05:25:43.021 ERROR:paramiko.transport:Traceback (most recent call last):
2024-04-18T05:25:43.021 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2143, in run
2024-04-18T05:25:43.021 ERROR:paramiko.transport:    self._check_banner()
2024-04-18T05:25:43.021 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2331, in _check_banner
2024-04-18T05:25:43.022 ERROR:paramiko.transport:    raise SSHException(
2024-04-18T05:25:43.022 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2024-04-18T05:25:43.022 ERROR:paramiko.transport:
2024-04-18T05:25:43.022 ERROR:teuthology.orchestra.connection:Error authenticating with smithi144.front.sepia.ceph.com: SSHException('Error reading SSH protocol banner') (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2327, in _check_banner
    buf = self.packetizer.readline(timeout)
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 381, in readline
    buf += self._read_timeout(timeout)
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 618, in _read_timeout
    raise EOFError()
EOFError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/connection.py", line 106, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 451, in connect
    t.start_client(timeout=timeout)
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 738, in start_client
    raise e
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2143, in run
    self._check_banner()
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2331, in _check_banner
    raise SSHException(
paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2024-04-18T05:25:45.203 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:25:48.254 WARNING:teuthology.provision.fog.smithi100:[Errno None] Unable to connect to port 22 on 172.21.15.100
2024-04-18T05:25:54.255 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:25:57.310 WARNING:teuthology.provision.fog.smithi100:[Errno None] Unable to connect to port 22 on 172.21.15.100
2024-04-18T05:26:03.312 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:26:05.107 ERROR:teuthology.orchestra.connection:Error authenticating with smithi144.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-04-18T05:26:06.366 WARNING:teuthology.provision.fog.smithi100:[Errno None] Unable to connect to port 22 on 172.21.15.100
2024-04-18T05:26:12.368 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:26:15.422 WARNING:teuthology.provision.fog.smithi100:[Errno None] Unable to connect to port 22 on 172.21.15.100
2024-04-18T05:26:21.424 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:26:24.482 WARNING:teuthology.provision.fog.smithi100:[Errno None] Unable to connect to port 22 on 172.21.15.100
2024-04-18T05:26:30.483 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:26:33.534 WARNING:teuthology.provision.fog.smithi100:[Errno None] Unable to connect to port 22 on 172.21.15.100
2024-04-18T05:26:39.536 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:26:58.078 WARNING:teuthology.provision.fog.smithi100:[Errno None] Unable to connect to port 22 on 172.21.15.100
2024-04-18T05:27:04.080 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:27:07.138 WARNING:teuthology.provision.fog.smithi100:[Errno None] Unable to connect to port 22 on 172.21.15.100
2024-04-18T05:27:13.140 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:27:16.190 WARNING:teuthology.provision.fog.smithi100:[Errno None] Unable to connect to port 22 on 172.21.15.100
2024-04-18T05:27:22.191 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:27:23.200 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-04-18T05:27:23.201 DEBUG:teuthology.orchestra.run.smithi100:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-04-18T05:27:30.109 WARNING:teuthology.provision.fog.smithi144:timed out
2024-04-18T05:27:36.111 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:27:38.509 INFO:teuthology.provision.fog.smithi100:Node is ready
2024-04-18T05:27:38.509 DEBUG:teuthology.orchestra.run.smithi100:> hostname
2024-04-18T05:27:38.531 INFO:teuthology.orchestra.run.smithi100.stdout:smithi100.front.sepia.ceph.com
2024-04-18T05:27:38.531 DEBUG:teuthology.orchestra.run.smithi100:> grep smithi100.front.sepia.ceph.com /etc/hosts
2024-04-18T05:27:38.592 INFO:teuthology.orchestra.run.smithi100.stdout:172.21.15.100 smithi100.front.sepia.ceph.com smithi100
2024-04-18T05:27:38.592 DEBUG:teuthology.orchestra.run.smithi100:> sudo hostname smithi100
2024-04-18T05:27:38.721 DEBUG:teuthology.orchestra.run.smithi100:> sudo sed -i -e 's/smithi100.front.sepia.ceph.com/smithi100/g' /etc/hosts
2024-04-18T05:27:38.751 DEBUG:teuthology.orchestra.run.smithi100:> sudo sed -i -e 's/172.21.15.100/172.21.15.100/g' /etc/hosts
2024-04-18T05:27:38.820 DEBUG:teuthology.orchestra.run.smithi100:> sudo sed -i -e 's/smithi100.front.sepia.ceph.com/smithi100/g' /etc/hostname
2024-04-18T05:27:38.889 DEBUG:teuthology.orchestra.run.smithi100:> sudo hostname smithi100
2024-04-18T05:27:38.956 DEBUG:teuthology.orchestra.run.smithi100:> cat /etc/os-release
2024-04-18T05:27:39.014 INFO:teuthology.orchestra.run.smithi100.stdout:NAME="Red Hat Enterprise Linux"
2024-04-18T05:27:39.014 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION="8.6 (Ootpa)"
2024-04-18T05:27:39.014 INFO:teuthology.orchestra.run.smithi100.stdout:ID="rhel"
2024-04-18T05:27:39.014 INFO:teuthology.orchestra.run.smithi100.stdout:ID_LIKE="fedora"
2024-04-18T05:27:39.014 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION_ID="8.6"
2024-04-18T05:27:39.014 INFO:teuthology.orchestra.run.smithi100.stdout:PLATFORM_ID="platform:el8"
2024-04-18T05:27:39.014 INFO:teuthology.orchestra.run.smithi100.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2024-04-18T05:27:39.014 INFO:teuthology.orchestra.run.smithi100.stdout:ANSI_COLOR="0;31"
2024-04-18T05:27:39.015 INFO:teuthology.orchestra.run.smithi100.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2024-04-18T05:27:39.015 INFO:teuthology.orchestra.run.smithi100.stdout:HOME_URL="https://www.redhat.com/"
2024-04-18T05:27:39.015 INFO:teuthology.orchestra.run.smithi100.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2024-04-18T05:27:39.015 INFO:teuthology.orchestra.run.smithi100.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-04-18T05:27:39.015 INFO:teuthology.orchestra.run.smithi100.stdout:
2024-04-18T05:27:39.015 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2024-04-18T05:27:39.015 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2024-04-18T05:27:39.015 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2024-04-18T05:27:39.015 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2024-04-18T05:27:39.016 INFO:teuthology.provision.fog.smithi100:Deploy complete!
2024-04-18T05:27:39.017 DEBUG:teuthology.parallel:result is None
2024-04-18T05:27:40.190 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144
2024-04-18T05:27:41.099 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:27:44.158 WARNING:teuthology.provision.fog.smithi077:[Errno None] Unable to connect to port 22 on 172.21.15.77
2024-04-18T05:27:46.192 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:27:46.974 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144
2024-04-18T05:27:50.159 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:27:52.976 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:27:53.214 WARNING:teuthology.provision.fog.smithi077:[Errno None] Unable to connect to port 22 on 172.21.15.77
2024-04-18T05:27:59.216 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:28:01.192 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-04-18T05:28:01.192 DEBUG:teuthology.orchestra.run.smithi144:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-04-18T05:28:02.270 WARNING:teuthology.provision.fog.smithi077:[Errno None] Unable to connect to port 22 on 172.21.15.77
2024-04-18T05:28:08.271 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:28:11.326 WARNING:teuthology.provision.fog.smithi077:[Errno None] Unable to connect to port 22 on 172.21.15.77
2024-04-18T05:28:17.328 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:28:20.382 WARNING:teuthology.provision.fog.smithi077:[Errno None] Unable to connect to port 22 on 172.21.15.77
2024-04-18T05:28:21.655 INFO:teuthology.provision.fog.smithi144:Node is ready
2024-04-18T05:28:21.655 DEBUG:teuthology.orchestra.run.smithi144:> hostname
2024-04-18T05:28:21.676 INFO:teuthology.orchestra.run.smithi144.stdout:smithi144.front.sepia.ceph.com
2024-04-18T05:28:21.677 DEBUG:teuthology.orchestra.run.smithi144:> grep smithi144.front.sepia.ceph.com /etc/hosts
2024-04-18T05:28:21.735 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144 smithi144.front.sepia.ceph.com smithi144
2024-04-18T05:28:21.735 DEBUG:teuthology.orchestra.run.smithi144:> sudo hostname smithi144
2024-04-18T05:28:21.892 DEBUG:teuthology.orchestra.run.smithi144:> sudo sed -i -e 's/smithi144.front.sepia.ceph.com/smithi144/g' /etc/hosts
2024-04-18T05:28:21.928 DEBUG:teuthology.orchestra.run.smithi144:> sudo sed -i -e 's/172.21.15.144/172.21.15.144/g' /etc/hosts
2024-04-18T05:28:21.996 DEBUG:teuthology.orchestra.run.smithi144:> sudo sed -i -e 's/smithi144.front.sepia.ceph.com/smithi144/g' /etc/hostname
2024-04-18T05:28:22.063 DEBUG:teuthology.orchestra.run.smithi144:> sudo hostname smithi144
2024-04-18T05:28:22.130 DEBUG:teuthology.orchestra.run.smithi144:> cat /etc/os-release
2024-04-18T05:28:22.186 INFO:teuthology.orchestra.run.smithi144.stdout:NAME="Red Hat Enterprise Linux"
2024-04-18T05:28:22.186 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION="8.6 (Ootpa)"
2024-04-18T05:28:22.186 INFO:teuthology.orchestra.run.smithi144.stdout:ID="rhel"
2024-04-18T05:28:22.186 INFO:teuthology.orchestra.run.smithi144.stdout:ID_LIKE="fedora"
2024-04-18T05:28:22.186 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION_ID="8.6"
2024-04-18T05:28:22.186 INFO:teuthology.orchestra.run.smithi144.stdout:PLATFORM_ID="platform:el8"
2024-04-18T05:28:22.186 INFO:teuthology.orchestra.run.smithi144.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2024-04-18T05:28:22.186 INFO:teuthology.orchestra.run.smithi144.stdout:ANSI_COLOR="0;31"
2024-04-18T05:28:22.186 INFO:teuthology.orchestra.run.smithi144.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2024-04-18T05:28:22.187 INFO:teuthology.orchestra.run.smithi144.stdout:HOME_URL="https://www.redhat.com/"
2024-04-18T05:28:22.187 INFO:teuthology.orchestra.run.smithi144.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2024-04-18T05:28:22.187 INFO:teuthology.orchestra.run.smithi144.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-04-18T05:28:22.187 INFO:teuthology.orchestra.run.smithi144.stdout:
2024-04-18T05:28:22.187 INFO:teuthology.orchestra.run.smithi144.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2024-04-18T05:28:22.187 INFO:teuthology.orchestra.run.smithi144.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2024-04-18T05:28:22.187 INFO:teuthology.orchestra.run.smithi144.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2024-04-18T05:28:22.187 INFO:teuthology.orchestra.run.smithi144.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2024-04-18T05:28:22.187 INFO:teuthology.provision.fog.smithi144:Deploy complete!
2024-04-18T05:28:22.188 DEBUG:teuthology.parallel:result is None
2024-04-18T05:28:26.384 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:28:29.438 WARNING:teuthology.provision.fog.smithi077:[Errno None] Unable to connect to port 22 on 172.21.15.77
2024-04-18T05:28:35.440 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:29:35.442 WARNING:teuthology.provision.fog.smithi077:timed out
2024-04-18T05:29:41.444 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T05:29:41.693 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-04-18T05:29:41.693 DEBUG:teuthology.orchestra.run.smithi077:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-04-18T05:29:41.940 INFO:teuthology.provision.fog.smithi077:Node is ready
2024-04-18T05:29:41.940 DEBUG:teuthology.orchestra.run.smithi077:> hostname
2024-04-18T05:29:41.958 INFO:teuthology.orchestra.run.smithi077.stdout:smithi077
2024-04-18T05:29:41.959 DEBUG:teuthology.orchestra.run.smithi077:> grep smithi077 /etc/hosts
2024-04-18T05:29:42.017 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77 smithi077.front.sepia.ceph.com smithi077
2024-04-18T05:29:42.017 DEBUG:teuthology.orchestra.run.smithi077:> sudo hostname smithi077
2024-04-18T05:29:42.138 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i -e 's/smithi077/smithi077/g' /etc/hosts
2024-04-18T05:29:42.168 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i -e 's/172.21.15.77/172.21.15.77/g' /etc/hosts
2024-04-18T05:29:42.236 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i -e 's/smithi077/smithi077/g' /etc/hostname
2024-04-18T05:29:42.305 DEBUG:teuthology.orchestra.run.smithi077:> sudo hostname smithi077
2024-04-18T05:29:42.371 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release
2024-04-18T05:29:42.428 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="Red Hat Enterprise Linux"
2024-04-18T05:29:42.428 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="8.6 (Ootpa)"
2024-04-18T05:29:42.428 INFO:teuthology.orchestra.run.smithi077.stdout:ID="rhel"
2024-04-18T05:29:42.428 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE="fedora"
2024-04-18T05:29:42.429 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="8.6"
2024-04-18T05:29:42.429 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM_ID="platform:el8"
2024-04-18T05:29:42.429 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2024-04-18T05:29:42.429 INFO:teuthology.orchestra.run.smithi077.stdout:ANSI_COLOR="0;31"
2024-04-18T05:29:42.429 INFO:teuthology.orchestra.run.smithi077.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2024-04-18T05:29:42.429 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://www.redhat.com/"
2024-04-18T05:29:42.429 INFO:teuthology.orchestra.run.smithi077.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2024-04-18T05:29:42.429 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-04-18T05:29:42.429 INFO:teuthology.orchestra.run.smithi077.stdout:
2024-04-18T05:29:42.429 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2024-04-18T05:29:42.429 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2024-04-18T05:29:42.429 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2024-04-18T05:29:42.429 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2024-04-18T05:29:42.430 INFO:teuthology.provision.fog.smithi077:Deploy complete!
2024-04-18T05:29:42.430 DEBUG:teuthology.parallel:result is None
2024-04-18T05:29:43.625 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/lock/ops.py", line 326, in reimage_machines
    reimaged = do_update_keys(list(reimaged.keys()))[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/lock/ops.py", line 290, in do_update_keys
    keys_dict = misc.ssh_keyscan(machines, _raise=_raise)
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/misc.py", line 1108, in ssh_keyscan
    key = _ssh_keyscan(hostname)
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/misc.py", line 1145, in _ssh_keyscan
    return sorted(keys)[0]
IndexError: list index out of range
2024-04-18T05:29:43.627 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-04-18T05:29:43.914 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=4115091bce9e46a7a3e7a43fbf67ef74
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/lock/ops.py", line 326, in reimage_machines
    reimaged = do_update_keys(list(reimaged.keys()))[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/lock/ops.py", line 290, in do_update_keys
    keys_dict = misc.ssh_keyscan(machines, _raise=_raise)
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/misc.py", line 1108, in ssh_keyscan
    key = _ssh_keyscan(hostname)
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/misc.py", line 1145, in _ssh_keyscan
    return sorted(keys)[0]
IndexError: list index out of range
2024-04-18T05:29:43.916 INFO:teuthology.nuke:Checking targets against current locks
2024-04-18T05:29:43.973 DEBUG:teuthology.nuke:shortname: smithi077
2024-04-18T05:29:43.974 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-04-18T05:29:44.001 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-17_22:24:03-fs-reef-distro-default-smithi/7661580', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-18 05:22:42.845512', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEl09AZ15EIANNtbUpswZiJy+8qGPJFcSe9J+WZ3qxUohnLJP5NzMCy3z/SaKZ8DxdIEHQgG4SVCtDtuN6j3IgE='}
2024-04-18T05:29:44.005 DEBUG:teuthology.nuke:shortname: smithi100
2024-04-18T05:29:44.005 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-04-18T05:29:44.027 DEBUG:teuthology.nuke:shortname: smithi144
2024-04-18T05:29:44.027 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-04-18T05:29:44.031 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi100.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-17_22:24:03-fs-reef-distro-default-smithi/7661580', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-18 05:22:42.841421', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF47Ixs1xYmyO3bkBsO55DEOF1WTWTzvezRD59XLBjjya6w3zY+dKiS/nF1U2R2MiYSZr3knkRx8jopaRp3gnbM='}
2024-04-18T05:29:44.056 INFO:teuthology.orchestra.console.smithi077:Power off
2024-04-18T05:29:44.056 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-18T05:29:44.067 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi144.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-17_22:24:03-fs-reef-distro-default-smithi/7661595', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-18 05:28:39.850967', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJs1YzjIvzIKFcTOK69LTIzQh6EztOwE4KmE4SChrEYBFnclcwFrw/GIjVT4BKjdOIJbXtKkJoOz3Gg+uHu6mhU='}
2024-04-18T05:29:44.079 DEBUG:teuthology.orchestra.console.smithi077:power off output: Chassis Power Control: Down/Off
2024-04-18T05:29:44.079 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T05:29:44.102 INFO:teuthology.orchestra.console.smithi100:Power off
2024-04-18T05:29:44.103 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-18T05:29:44.111 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on
2024-04-18T05:29:44.124 DEBUG:teuthology.orchestra.console.smithi100:power off output: Chassis Power Control: Down/Off
2024-04-18T05:29:44.124 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T05:29:44.139 INFO:teuthology.orchestra.console.smithi144:Power off
2024-04-18T05:29:44.139 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-18T05:29:44.150 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is on
2024-04-18T05:29:44.162 DEBUG:teuthology.orchestra.console.smithi144:power off output: Chassis Power Control: Down/Off
2024-04-18T05:29:44.162 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T05:29:44.182 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is on
2024-04-18T05:29:48.113 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T05:29:48.151 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T05:29:48.183 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T05:29:48.224 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on
2024-04-18T05:29:48.262 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is on
2024-04-18T05:29:48.295 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is off
2024-04-18T05:29:48.396 INFO:teuthology.orchestra.console.smithi144:Power off completed
2024-04-18T05:29:48.546 INFO:teuthology.lock.ops:unlocked: smithi144.front.sepia.ceph.com
2024-04-18T05:29:52.225 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T05:29:52.263 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T05:29:52.335 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off
2024-04-18T05:29:52.374 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is on
2024-04-18T05:29:52.436 INFO:teuthology.orchestra.console.smithi077:Power off completed
2024-04-18T05:29:52.586 INFO:teuthology.lock.ops:unlocked: smithi077.front.sepia.ceph.com
2024-04-18T05:29:56.375 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T05:29:56.485 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is off
2024-04-18T05:29:56.586 INFO:teuthology.orchestra.console.smithi100:Power off completed
2024-04-18T05:29:56.729 INFO:teuthology.lock.ops:unlocked: smithi100.front.sepia.ceph.com
2024-04-18T05:29:57.103 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/lock/ops.py", line 326, in reimage_machines
    reimaged = do_update_keys(list(reimaged.keys()))[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/lock/ops.py", line 290, in do_update_keys
    keys_dict = misc.ssh_keyscan(machines, _raise=_raise)
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/misc.py", line 1108, in ssh_keyscan
    key = _ssh_keyscan(hostname)
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/misc.py", line 1145, in _ssh_keyscan
    return sorted(keys)[0]
IndexError: list index out of range