2024-02-16T11:05:37.125 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55
2024-02-16T11:05:37.126 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-02-16T11:05:37.315 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f841e637f10>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f841f697c10>, signals=[15])
2024-02-16T11:05:37.316 INFO:teuthology.lock.ops:Start node 'smithi084.front.sepia.ceph.com' reimaging
2024-02-16T11:05:37.316 INFO:teuthology.lock.ops:Updating [smithi084.front.sepia.ceph.com]: reset os type and version on server
2024-02-16T11:05:37.316 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server
2024-02-16T11:05:37.347 INFO:teuthology.lock.ops:Start node 'smithi181.front.sepia.ceph.com' reimaging
2024-02-16T11:05:37.348 INFO:teuthology.lock.ops:Updating [smithi181.front.sepia.ceph.com]: reset os type and version on server
2024-02-16T11:05:37.348 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server
2024-02-16T11:05:37.373 INFO:teuthology.lock.ops:Start node 'smithi191.front.sepia.ceph.com' reimaging
2024-02-16T11:05:37.373 INFO:teuthology.lock.ops:Updating [smithi191.front.sepia.ceph.com]: reset os type and version on server
2024-02-16T11:05:37.373 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server
2024-02-16T11:05:37.452 DEBUG:teuthology.provision.fog.smithi084:Requesting image smithi_centos_9.stream (ID 10)
2024-02-16T11:05:37.490 INFO:teuthology.provision.fog.smithi084:Scheduling deploy of centos 9.stream
2024-02-16T11:05:37.498 DEBUG:teuthology.provision.fog.smithi181:Requesting image smithi_centos_9.stream (ID 10)
2024-02-16T11:05:37.513 DEBUG:teuthology.provision.fog.smithi191:Requesting image smithi_centos_9.stream (ID 10)
2024-02-16T11:05:37.546 INFO:teuthology.provision.fog.smithi181:Scheduling deploy of centos 9.stream
2024-02-16T11:05:37.560 INFO:teuthology.provision.fog.smithi191:Scheduling deploy of centos 9.stream
2024-02-16T11:05:37.661 DEBUG:teuthology.provision.fog.smithi191:Canceling deploy task with ID 796420
2024-02-16T11:05:37.903 INFO:teuthology.orchestra.console.smithi084:Power off
2024-02-16T11:05:37.903 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-02-16T11:05:37.917 INFO:teuthology.orchestra.console.smithi181:Power off
2024-02-16T11:05:37.917 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-02-16T11:05:37.928 DEBUG:teuthology.orchestra.console.smithi084:power off output: Chassis Power Control: Down/Off
2024-02-16T11:05:37.928 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-02-16T11:05:37.944 DEBUG:teuthology.orchestra.console.smithi181:power off output: Chassis Power Control: Down/Off
2024-02-16T11:05:37.944 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-02-16T11:05:37.954 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is off
2024-02-16T11:05:37.966 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is off
2024-02-16T11:05:37.973 INFO:teuthology.orchestra.console.smithi191:Power off
2024-02-16T11:05:37.973 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-02-16T11:05:37.998 DEBUG:teuthology.orchestra.console.smithi191:power off output: Chassis Power Control: Down/Off
2024-02-16T11:05:37.999 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-02-16T11:05:38.024 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is off
2024-02-16T11:05:38.056 INFO:teuthology.orchestra.console.smithi084:Power off completed
2024-02-16T11:05:38.067 INFO:teuthology.orchestra.console.smithi181:Power off completed
2024-02-16T11:05:38.125 INFO:teuthology.orchestra.console.smithi191:Power off completed
2024-02-16T11:05:38.157 INFO:teuthology.orchestra.console.smithi084:Power on
2024-02-16T11:05:38.157 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-02-16T11:05:38.168 INFO:teuthology.orchestra.console.smithi181:Power on
2024-02-16T11:05:38.168 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-02-16T11:05:38.179 DEBUG:teuthology.orchestra.console.smithi084:power on output: Chassis Power Control: Up/On
2024-02-16T11:05:38.179 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-02-16T11:05:38.194 DEBUG:teuthology.orchestra.console.smithi181:power on output: Chassis Power Control: Up/On
2024-02-16T11:05:38.194 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-02-16T11:05:38.209 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is off
2024-02-16T11:05:38.218 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is off
2024-02-16T11:05:38.226 INFO:teuthology.orchestra.console.smithi191:Power on
2024-02-16T11:05:38.227 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-02-16T11:05:38.256 DEBUG:teuthology.orchestra.console.smithi191:power on output: Chassis Power Control: Up/On
2024-02-16T11:05:38.257 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-02-16T11:05:38.290 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is off
2024-02-16T11:05:42.212 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-02-16T11:05:42.220 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-02-16T11:05:42.291 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-02-16T11:05:42.322 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is off
2024-02-16T11:05:42.330 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is off
2024-02-16T11:05:42.401 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is off
2024-02-16T11:05:46.324 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-02-16T11:05:46.331 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-02-16T11:05:46.402 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-02-16T11:05:46.433 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on
2024-02-16T11:05:46.442 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is on
2024-02-16T11:05:46.511 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on
2024-02-16T11:05:46.535 INFO:teuthology.orchestra.console.smithi084:Power on completed
2024-02-16T11:05:46.543 INFO:teuthology.orchestra.console.smithi181:Power on completed
2024-02-16T11:05:46.612 INFO:teuthology.orchestra.console.smithi191:Power on completed
2024-02-16T11:05:46.636 INFO:teuthology.provision.fog.smithi084:Waiting for deploy to finish
2024-02-16T11:05:46.644 INFO:teuthology.provision.fog.smithi181:Waiting for deploy to finish
2024-02-16T11:05:46.713 INFO:teuthology.provision.fog.smithi191:Waiting for deploy to finish
2024-02-16T11:05:46.756 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:05:46.822 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:05:49.308 WARNING:teuthology.provision.fog.smithi191:[Errno None] Unable to connect to port 22 on 172.21.15.191
2024-02-16T11:05:55.312 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:05:58.368 WARNING:teuthology.provision.fog.smithi191:[Errno None] Unable to connect to port 22 on 172.21.15.191
2024-02-16T11:06:04.372 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:06:05.244 WARNING:teuthology.provision.fog.smithi181:[Errno None] Unable to connect to port 22 on 172.21.15.181
2024-02-16T11:06:07.452 WARNING:teuthology.provision.fog.smithi191:[Errno None] Unable to connect to port 22 on 172.21.15.191
2024-02-16T11:06:11.248 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:06:13.456 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:06:14.304 WARNING:teuthology.provision.fog.smithi181:[Errno None] Unable to connect to port 22 on 172.21.15.181
2024-02-16T11:06:16.512 WARNING:teuthology.provision.fog.smithi191:[Errno None] Unable to connect to port 22 on 172.21.15.191
2024-02-16T11:06:20.308 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:06:22.514 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:06:23.388 WARNING:teuthology.provision.fog.smithi181:[Errno None] Unable to connect to port 22 on 172.21.15.181
2024-02-16T11:06:25.568 WARNING:teuthology.provision.fog.smithi191:[Errno None] Unable to connect to port 22 on 172.21.15.191
2024-02-16T11:06:29.392 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:06:31.571 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:06:32.444 WARNING:teuthology.provision.fog.smithi181:[Errno None] Unable to connect to port 22 on 172.21.15.181
2024-02-16T11:06:34.652 WARNING:teuthology.provision.fog.smithi191:[Errno None] Unable to connect to port 22 on 172.21.15.191
2024-02-16T11:06:38.448 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:06:40.654 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:06:41.500 WARNING:teuthology.provision.fog.smithi181:[Errno None] Unable to connect to port 22 on 172.21.15.181
2024-02-16T11:06:43.708 WARNING:teuthology.provision.fog.smithi191:[Errno None] Unable to connect to port 22 on 172.21.15.191
2024-02-16T11:06:47.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:06:49.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:06:50.556 WARNING:teuthology.provision.fog.smithi181:[Errno None] Unable to connect to port 22 on 172.21.15.181
2024-02-16T11:06:52.764 WARNING:teuthology.provision.fog.smithi191:[Errno None] Unable to connect to port 22 on 172.21.15.191
2024-02-16T11:06:56.560 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:06:58.768 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:07:14.876 WARNING:teuthology.provision.fog.smithi181:[Errno None] Unable to connect to port 22 on 172.21.15.181
2024-02-16T11:07:17.184 WARNING:teuthology.provision.fog.smithi191:[Errno None] Unable to connect to port 22 on 172.21.15.191
2024-02-16T11:07:20.880 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:07:23.187 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:07:23.932 WARNING:teuthology.provision.fog.smithi181:[Errno None] Unable to connect to port 22 on 172.21.15.181
2024-02-16T11:07:26.061 WARNING:teuthology.provision.fog.smithi191:[Errno None] Unable to connect to port 22 on 172.21.15.191
2024-02-16T11:07:29.938 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:07:30.079 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-02-16T11:07:32.063 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:07:32.199 ERROR:teuthology.orchestra.connection:Error authenticating with smithi191.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-02-16T11:07:34.215 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-02-16T11:07:36.337 ERROR:teuthology.orchestra.connection:Error authenticating with smithi191.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-02-16T11:07:41.350 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-02-16T11:07:43.441 ERROR:teuthology.orchestra.connection:Error authenticating with smithi191.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-02-16T11:07:51.479 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-02-16T11:07:53.561 ERROR:teuthology.orchestra.connection:Error authenticating with smithi191.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-02-16T11:08:04.611 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-02-16T11:08:06.709 ERROR:teuthology.orchestra.connection:Error authenticating with smithi191.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-02-16T11:08:18.085 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:08:20.742 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-02-16T11:08:21.148 WARNING:teuthology.provision.fog.smithi084:[Errno None] Unable to connect to port 22 on 172.21.15.84
2024-02-16T11:08:22.712 ERROR:paramiko.transport:Exception (client): Error reading SSH protocol banner
2024-02-16T11:08:22.754 ERROR:paramiko.transport:Traceback (most recent call last):
2024-02-16T11:08:22.754 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2327, in _check_banner
2024-02-16T11:08:22.754 ERROR:paramiko.transport:    buf = self.packetizer.readline(timeout)
2024-02-16T11:08:22.754 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 381, in readline
2024-02-16T11:08:22.754 ERROR:paramiko.transport:    buf += self._read_timeout(timeout)
2024-02-16T11:08:22.754 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 618, in _read_timeout
2024-02-16T11:08:22.754 ERROR:paramiko.transport:    raise EOFError()
2024-02-16T11:08:22.754 ERROR:paramiko.transport:EOFError
2024-02-16T11:08:22.754 ERROR:paramiko.transport:
2024-02-16T11:08:22.755 ERROR:paramiko.transport:During handling of the above exception, another exception occurred:
2024-02-16T11:08:22.755 ERROR:paramiko.transport:
2024-02-16T11:08:22.755 ERROR:paramiko.transport:Traceback (most recent call last):
2024-02-16T11:08:22.755 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2143, in run
2024-02-16T11:08:22.755 ERROR:paramiko.transport:    self._check_banner()
2024-02-16T11:08:22.755 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2331, in _check_banner
2024-02-16T11:08:22.755 ERROR:paramiko.transport:    raise SSHException(
2024-02-16T11:08:22.755 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2024-02-16T11:08:22.755 ERROR:paramiko.transport:
2024-02-16T11:08:22.755 ERROR:teuthology.orchestra.connection:Error authenticating with smithi191.front.sepia.ceph.com: SSHException('Error reading SSH protocol banner') (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2327, in _check_banner
    buf = self.packetizer.readline(timeout)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 381, in readline
    buf += self._read_timeout(timeout)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 618, in _read_timeout
    raise EOFError()
EOFError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/connection.py", line 106, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 451, in connect
    t.start_client(timeout=timeout)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 738, in start_client
    raise e
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2143, in run
    self._check_banner()
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2331, in _check_banner
    raise SSHException(
paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2024-02-16T11:08:27.149 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:08:30.204 WARNING:teuthology.provision.fog.smithi084:[Errno None] Unable to connect to port 22 on 172.21.15.84
2024-02-16T11:08:36.205 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:08:39.264 WARNING:teuthology.provision.fog.smithi084:[Errno None] Unable to connect to port 22 on 172.21.15.84
2024-02-16T11:08:39.876 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-02-16T11:08:41.851 ERROR:teuthology.orchestra.connection:Error authenticating with smithi191.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-02-16T11:08:45.266 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:08:48.316 WARNING:teuthology.provision.fog.smithi084:[Errno None] Unable to connect to port 22 on 172.21.15.84
2024-02-16T11:08:54.317 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:08:57.376 WARNING:teuthology.provision.fog.smithi084:[Errno None] Unable to connect to port 22 on 172.21.15.84
2024-02-16T11:09:01.976 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-02-16T11:09:03.377 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:09:04.345 ERROR:teuthology.orchestra.connection:Error authenticating with smithi191.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-02-16T11:09:06.428 WARNING:teuthology.provision.fog.smithi084:[Errno None] Unable to connect to port 22 on 172.21.15.84
2024-02-16T11:09:12.430 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:09:15.484 WARNING:teuthology.provision.fog.smithi084:[Errno None] Unable to connect to port 22 on 172.21.15.84
2024-02-16T11:09:21.486 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:09:27.105 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-02-16T11:09:35.452 WARNING:teuthology.provision.fog.smithi191:[Errno None] Unable to connect to port 22 on 172.21.15.191
2024-02-16T11:09:41.454 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:09:44.508 WARNING:teuthology.provision.fog.smithi191:[Errno None] Unable to connect to port 22 on 172.21.15.191
2024-02-16T11:09:50.509 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:09:53.568 WARNING:teuthology.provision.fog.smithi191:[Errno None] Unable to connect to port 22 on 172.21.15.191
2024-02-16T11:09:59.570 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:10:01.212 WARNING:teuthology.provision.fog.smithi181:[Errno None] Unable to connect to port 22 on 172.21.15.181
2024-02-16T11:10:02.624 WARNING:teuthology.provision.fog.smithi191:[Errno None] Unable to connect to port 22 on 172.21.15.191
2024-02-16T11:10:07.214 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:10:07.676 WARNING:teuthology.provision.fog.smithi181:[Errno None] Unable to connect to port 22 on 172.21.15.181
2024-02-16T11:10:08.626 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:10:13.678 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:10:16.736 WARNING:teuthology.provision.fog.smithi181:[Errno None] Unable to connect to port 22 on 172.21.15.181
2024-02-16T11:10:21.488 WARNING:teuthology.provision.fog.smithi084:timed out
2024-02-16T11:10:22.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:10:25.792 WARNING:teuthology.provision.fog.smithi181:[Errno None] Unable to connect to port 22 on 172.21.15.181
2024-02-16T11:10:27.490 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:10:27.821 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-02-16T11:10:27.822 DEBUG:teuthology.orchestra.run.smithi084:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-02-16T11:10:28.150 INFO:teuthology.provision.fog.smithi084:Node is ready
2024-02-16T11:10:28.150 DEBUG:teuthology.orchestra.run.smithi084:> hostname
2024-02-16T11:10:28.164 INFO:teuthology.orchestra.run.smithi084.stdout:smithi084.front.sepia.ceph.com
2024-02-16T11:10:28.165 DEBUG:teuthology.orchestra.run.smithi084:> grep smithi084.front.sepia.ceph.com /etc/hosts
2024-02-16T11:10:28.220 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84 smithi084.front.sepia.ceph.com smithi084
2024-02-16T11:10:28.220 DEBUG:teuthology.orchestra.run.smithi084:> sudo hostname smithi084
2024-02-16T11:10:28.323 DEBUG:teuthology.orchestra.run.smithi084:> sudo sed -i -e 's/smithi084.front.sepia.ceph.com/smithi084/g' /etc/hosts
2024-02-16T11:10:28.350 DEBUG:teuthology.orchestra.run.smithi084:> sudo sed -i -e 's/172.21.15.84/172.21.15.84/g' /etc/hosts
2024-02-16T11:10:28.418 DEBUG:teuthology.orchestra.run.smithi084:> sudo sed -i -e 's/smithi084.front.sepia.ceph.com/smithi084/g' /etc/hostname
2024-02-16T11:10:28.483 DEBUG:teuthology.orchestra.run.smithi084:> sudo hostname smithi084
2024-02-16T11:10:28.547 DEBUG:teuthology.orchestra.run.smithi084:> cat /etc/os-release
2024-02-16T11:10:28.598 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="CentOS Stream"
2024-02-16T11:10:28.598 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="9"
2024-02-16T11:10:28.598 INFO:teuthology.orchestra.run.smithi084.stdout:ID="centos"
2024-02-16T11:10:28.598 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE="rhel fedora"
2024-02-16T11:10:28.598 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="9"
2024-02-16T11:10:28.598 INFO:teuthology.orchestra.run.smithi084.stdout:PLATFORM_ID="platform:el9"
2024-02-16T11:10:28.598 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="CentOS Stream 9"
2024-02-16T11:10:28.598 INFO:teuthology.orchestra.run.smithi084.stdout:ANSI_COLOR="0;31"
2024-02-16T11:10:28.598 INFO:teuthology.orchestra.run.smithi084.stdout:LOGO="fedora-logo-icon"
2024-02-16T11:10:28.599 INFO:teuthology.orchestra.run.smithi084.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-02-16T11:10:28.599 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://centos.org/"
2024-02-16T11:10:28.599 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-02-16T11:10:28.599 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-02-16T11:10:28.599 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-02-16T11:10:28.600 INFO:teuthology.provision.fog.smithi084:Deploy complete!
2024-02-16T11:10:28.601 DEBUG:teuthology.parallel:result is None
2024-02-16T11:10:31.793 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:10:34.848 WARNING:teuthology.provision.fog.smithi181:[Errno None] Unable to connect to port 22 on 172.21.15.181
2024-02-16T11:10:40.849 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:11:08.628 WARNING:teuthology.provision.fog.smithi191:timed out
2024-02-16T11:11:14.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:11:14.768 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-02-16T11:11:14.769 DEBUG:teuthology.orchestra.run.smithi191:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-02-16T11:11:14.837 INFO:teuthology.provision.fog.smithi191:Node is ready
2024-02-16T11:11:14.837 DEBUG:teuthology.orchestra.run.smithi191:> hostname
2024-02-16T11:11:14.891 INFO:teuthology.orchestra.run.smithi191.stdout:smithi191
2024-02-16T11:11:14.891 DEBUG:teuthology.orchestra.run.smithi191:> grep smithi191 /etc/hosts
2024-02-16T11:11:14.945 INFO:teuthology.orchestra.run.smithi191.stdout:172.21.15.191 smithi191 smithi191
2024-02-16T11:11:14.946 DEBUG:teuthology.orchestra.run.smithi191:> sudo hostname smithi191
2024-02-16T11:11:15.013 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i -e 's/smithi191/smithi191/g' /etc/hosts
2024-02-16T11:11:15.080 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i -e 's/172.21.15.191/172.21.15.191/g' /etc/hosts
2024-02-16T11:11:15.147 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i -e 's/smithi191/smithi191/g' /etc/hostname
2024-02-16T11:11:15.214 DEBUG:teuthology.orchestra.run.smithi191:> sudo hostname smithi191
2024-02-16T11:11:15.281 DEBUG:teuthology.orchestra.run.smithi191:> cat /etc/os-release
2024-02-16T11:11:15.335 INFO:teuthology.orchestra.run.smithi191.stdout:NAME="CentOS Stream"
2024-02-16T11:11:15.336 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION="9"
2024-02-16T11:11:15.336 INFO:teuthology.orchestra.run.smithi191.stdout:ID="centos"
2024-02-16T11:11:15.336 INFO:teuthology.orchestra.run.smithi191.stdout:ID_LIKE="rhel fedora"
2024-02-16T11:11:15.336 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_ID="9"
2024-02-16T11:11:15.336 INFO:teuthology.orchestra.run.smithi191.stdout:PLATFORM_ID="platform:el9"
2024-02-16T11:11:15.336 INFO:teuthology.orchestra.run.smithi191.stdout:PRETTY_NAME="CentOS Stream 9"
2024-02-16T11:11:15.336 INFO:teuthology.orchestra.run.smithi191.stdout:ANSI_COLOR="0;31"
2024-02-16T11:11:15.336 INFO:teuthology.orchestra.run.smithi191.stdout:LOGO="fedora-logo-icon"
2024-02-16T11:11:15.336 INFO:teuthology.orchestra.run.smithi191.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-02-16T11:11:15.336 INFO:teuthology.orchestra.run.smithi191.stdout:HOME_URL="https://centos.org/"
2024-02-16T11:11:15.336 INFO:teuthology.orchestra.run.smithi191.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-02-16T11:11:15.336 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-02-16T11:11:15.337 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-02-16T11:11:15.337 INFO:teuthology.provision.fog.smithi191:Deploy complete!
2024-02-16T11:11:15.338 DEBUG:teuthology.parallel:result is None
2024-02-16T11:11:40.851 WARNING:teuthology.provision.fog.smithi181:timed out
2024-02-16T11:11:46.854 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-16T11:11:47.077 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-02-16T11:11:47.078 DEBUG:teuthology.orchestra.run.smithi181:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-02-16T11:11:47.410 INFO:teuthology.provision.fog.smithi181:Node is ready
2024-02-16T11:11:47.411 DEBUG:teuthology.orchestra.run.smithi181:> hostname
2024-02-16T11:11:47.425 INFO:teuthology.orchestra.run.smithi181.stdout:smithi181.front.sepia.ceph.com
2024-02-16T11:11:47.425 DEBUG:teuthology.orchestra.run.smithi181:> grep smithi181.front.sepia.ceph.com /etc/hosts
2024-02-16T11:11:47.480 INFO:teuthology.orchestra.run.smithi181.stdout:172.21.15.181 smithi181.front.sepia.ceph.com smithi181
2024-02-16T11:11:47.481 DEBUG:teuthology.orchestra.run.smithi181:> sudo hostname smithi181
2024-02-16T11:11:47.588 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i -e 's/smithi181.front.sepia.ceph.com/smithi181/g' /etc/hosts
2024-02-16T11:11:47.616 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i -e 's/172.21.15.181/172.21.15.181/g' /etc/hosts
2024-02-16T11:11:47.684 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i -e 's/smithi181.front.sepia.ceph.com/smithi181/g' /etc/hostname
2024-02-16T11:11:47.750 DEBUG:teuthology.orchestra.run.smithi181:> sudo hostname smithi181
2024-02-16T11:11:47.817 DEBUG:teuthology.orchestra.run.smithi181:> cat /etc/os-release
2024-02-16T11:11:47.871 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="CentOS Stream"
2024-02-16T11:11:47.871 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="9"
2024-02-16T11:11:47.871 INFO:teuthology.orchestra.run.smithi181.stdout:ID="centos"
2024-02-16T11:11:47.871 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="rhel fedora"
2024-02-16T11:11:47.872 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="9"
2024-02-16T11:11:47.872 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el9"
2024-02-16T11:11:47.872 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="CentOS Stream 9"
2024-02-16T11:11:47.872 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31"
2024-02-16T11:11:47.872 INFO:teuthology.orchestra.run.smithi181.stdout:LOGO="fedora-logo-icon"
2024-02-16T11:11:47.872 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-02-16T11:11:47.872 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://centos.org/"
2024-02-16T11:11:47.872 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-02-16T11:11:47.872 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-02-16T11:11:47.872 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-02-16T11:11:47.873 INFO:teuthology.provision.fog.smithi181:Deploy complete!
2024-02-16T11:11:47.873 DEBUG:teuthology.parallel:result is None
2024-02-16T11:11:49.078 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/lock/ops.py", line 326, in reimage_machines
    reimaged = do_update_keys(list(reimaged.keys()))[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/lock/ops.py", line 290, in do_update_keys
    keys_dict = misc.ssh_keyscan(machines, _raise=_raise)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1108, in ssh_keyscan
    key = _ssh_keyscan(hostname)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1145, in _ssh_keyscan
    return sorted(keys)[0]
IndexError: list index out of range
2024-02-16T11:11:49.080 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-02-16T11:11:49.376 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=eafd46d435df472db531dd57330c507f
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/lock/ops.py", line 326, in reimage_machines
    reimaged = do_update_keys(list(reimaged.keys()))[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/lock/ops.py", line 290, in do_update_keys
    keys_dict = misc.ssh_keyscan(machines, _raise=_raise)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1108, in ssh_keyscan
    key = _ssh_keyscan(hostname)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1145, in _ssh_keyscan
    return sorted(keys)[0]
IndexError: list index out of range
2024-02-16T11:11:49.378 INFO:teuthology.nuke:Checking targets against current locks
2024-02-16T11:11:49.427 DEBUG:teuthology.nuke:shortname: smithi084
2024-02-16T11:11:49.428 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-02-16T11:11:49.455 DEBUG:teuthology.nuke:shortname: smithi181
2024-02-16T11:11:49.455 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-02-16T11:11:49.477 DEBUG:teuthology.nuke:shortname: smithi191
2024-02-16T11:11:49.477 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-02-16T11:11:49.481 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-02-16_10:35:24-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7562066', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-16 11:05:36.377641', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCRxUgT5/rnQxoaOty7sHVKjAFjQIPediiwt7lQCIFpzEvaMHQwp51ENHU2+FUPlIupSaKQr9oOJch+3wpNA5WM='}
2024-02-16T11:11:49.484 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi084.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-02-16_10:35:24-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7562065', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-16 11:05:35.958003', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMJ1C0u4NgEVp0gsj5POEx4XG2ldLp1tk7JlThzC5xYYWohnCPIZ7WnMQiYglcq8lV601zlyU9DIopjQOx/62ZU='}
2024-02-16T11:11:49.499 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-02-16_10:35:24-fs-wip-vshankar-testing-20240216.060239-testing-default-smithi/7562073', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-16 11:11:30.093632', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH/P1Ow6fV0tvSD6Lu4au7BorBHSWjMNrUdsY6qQO/YuXxHG2Mn7qQKfCtdEQadtLV39QgCEjhNjoaJbWd5mjkA='}
2024-02-16T11:11:49.529 INFO:teuthology.orchestra.console.smithi181:Power off
2024-02-16T11:11:49.530 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-02-16T11:11:49.545 INFO:teuthology.orchestra.console.smithi084:Power off
2024-02-16T11:11:49.546 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-02-16T11:11:49.555 DEBUG:teuthology.orchestra.console.smithi181:power off output: Chassis Power Control: Down/Off
2024-02-16T11:11:49.556 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-02-16T11:11:49.572 INFO:teuthology.orchestra.console.smithi191:Power off
2024-02-16T11:11:49.573 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-02-16T11:11:49.581 DEBUG:teuthology.orchestra.console.smithi084:power off output: Chassis Power Control: Down/Off
2024-02-16T11:11:49.582 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-02-16T11:11:49.591 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is on
2024-02-16T11:11:49.595 DEBUG:teuthology.orchestra.console.smithi191:power off output: Chassis Power Control: Down/Off
2024-02-16T11:11:49.595 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-02-16T11:11:49.607 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on
2024-02-16T11:11:49.618 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on
2024-02-16T11:11:53.593 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-02-16T11:11:53.609 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-02-16T11:11:53.619 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-02-16T11:11:53.703 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is on
2024-02-16T11:11:53.719 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on
2024-02-16T11:11:53.728 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on
2024-02-16T11:11:57.704 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-02-16T11:11:57.720 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-02-16T11:11:57.730 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-02-16T11:11:57.816 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is on
2024-02-16T11:11:57.830 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is off
2024-02-16T11:11:57.843 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is off
2024-02-16T11:11:57.932 INFO:teuthology.orchestra.console.smithi084:Power off completed
2024-02-16T11:11:57.944 INFO:teuthology.orchestra.console.smithi191:Power off completed
2024-02-16T11:11:58.080 INFO:teuthology.lock.ops:unlocked: smithi084.front.sepia.ceph.com
2024-02-16T11:11:58.081 ERROR:teuthology.lock.ops:failed to unlock smithi191.front.sepia.ceph.com. reason: Cannot unlock an already-unlocked node
2024-02-16T11:12:01.817 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-02-16T11:12:01.928 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is off
2024-02-16T11:12:02.030 INFO:teuthology.orchestra.console.smithi181:Power off completed
2024-02-16T11:12:02.171 ERROR:teuthology.lock.ops:failed to unlock smithi181.front.sepia.ceph.com. reason: Cannot unlock an already-unlocked node
2024-02-16T11:12:02.507 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/lock/ops.py", line 326, in reimage_machines
    reimaged = do_update_keys(list(reimaged.keys()))[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/lock/ops.py", line 290, in do_update_keys
    keys_dict = misc.ssh_keyscan(machines, _raise=_raise)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1108, in ssh_keyscan
    key = _ssh_keyscan(hostname)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 1145, in _ssh_keyscan
    return sorted(keys)[0]
IndexError: list index out of range