2024-04-14T16:10:18.977 INFO:root:teuthology version: 0.0.1.dev267+g6c63784
2024-04-14T16:10:18.978 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-04-14T16:10:19.181 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7fad3fb0bf40>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7fad40b6cc10>, signals=[15])
2024-04-14T16:10:19.181 INFO:teuthology.lock.ops:Start node 'smithi057.front.sepia.ceph.com' reimaging
2024-04-14T16:10:19.182 INFO:teuthology.lock.ops:Updating [smithi057.front.sepia.ceph.com]: reset os type and version on server
2024-04-14T16:10:19.182 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server
2024-04-14T16:10:19.205 INFO:teuthology.lock.ops:Start node 'smithi107.front.sepia.ceph.com' reimaging
2024-04-14T16:10:19.206 INFO:teuthology.lock.ops:Updating [smithi107.front.sepia.ceph.com]: reset os type and version on server
2024-04-14T16:10:19.206 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server
2024-04-14T16:10:19.230 INFO:teuthology.lock.ops:Start node 'smithi131.front.sepia.ceph.com' reimaging
2024-04-14T16:10:19.230 INFO:teuthology.lock.ops:Updating [smithi131.front.sepia.ceph.com]: reset os type and version on server
2024-04-14T16:10:19.231 INFO:teuthology.lock.ops:Updating smithi131.front.sepia.ceph.com on lock server
2024-04-14T16:10:19.255 INFO:teuthology.lock.ops:Start node 'smithi150.front.sepia.ceph.com' reimaging
2024-04-14T16:10:19.255 INFO:teuthology.lock.ops:Updating [smithi150.front.sepia.ceph.com]: reset os type and version on server
2024-04-14T16:10:19.255 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server
2024-04-14T16:10:19.326 DEBUG:teuthology.provision.fog.smithi057:Requesting image smithi_centos_9.stream (ID 10)
2024-04-14T16:10:19.332 DEBUG:teuthology.provision.fog.smithi107:Requesting image smithi_centos_9.stream (ID 10)
2024-04-14T16:10:19.366 INFO:teuthology.provision.fog.smithi107:Scheduling deploy of centos 9.stream
2024-04-14T16:10:19.370 DEBUG:teuthology.provision.fog.smithi131:Requesting image smithi_centos_9.stream (ID 10)
2024-04-14T16:10:19.371 INFO:teuthology.provision.fog.smithi057:Scheduling deploy of centos 9.stream
2024-04-14T16:10:19.380 DEBUG:teuthology.provision.fog.smithi150:Requesting image smithi_centos_9.stream (ID 10)
2024-04-14T16:10:19.414 INFO:teuthology.provision.fog.smithi131:Scheduling deploy of centos 9.stream
2024-04-14T16:10:19.433 INFO:teuthology.provision.fog.smithi150:Scheduling deploy of centos 9.stream
2024-04-14T16:10:19.594 INFO:teuthology.orchestra.console.smithi057:Power off
2024-04-14T16:10:19.594 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-14T16:10:19.610 INFO:teuthology.orchestra.console.smithi107:Power off
2024-04-14T16:10:19.611 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-14T16:10:19.624 INFO:teuthology.orchestra.console.smithi131:Power off
2024-04-14T16:10:19.624 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-14T16:10:19.638 INFO:teuthology.orchestra.console.smithi150:Power off
2024-04-14T16:10:19.639 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-14T16:10:19.646 DEBUG:teuthology.orchestra.console.smithi107:power off output: Chassis Power Control: Down/Off
2024-04-14T16:10:19.646 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-14T16:10:19.653 DEBUG:teuthology.orchestra.console.smithi057:power off output: Chassis Power Control: Down/Off
2024-04-14T16:10:19.654 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-14T16:10:19.662 DEBUG:teuthology.orchestra.console.smithi131:power off output: Chassis Power Control: Down/Off
2024-04-14T16:10:19.662 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-14T16:10:19.668 DEBUG:teuthology.orchestra.console.smithi150:power off output: Chassis Power Control: Down/Off
2024-04-14T16:10:19.669 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-14T16:10:19.675 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off
2024-04-14T16:10:19.677 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is off
2024-04-14T16:10:19.682 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is off
2024-04-14T16:10:19.688 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is off
2024-04-14T16:10:19.777 INFO:teuthology.orchestra.console.smithi107:Power off completed
2024-04-14T16:10:19.778 INFO:teuthology.orchestra.console.smithi057:Power off completed
2024-04-14T16:10:19.783 INFO:teuthology.orchestra.console.smithi131:Power off completed
2024-04-14T16:10:19.789 INFO:teuthology.orchestra.console.smithi150:Power off completed
2024-04-14T16:10:19.877 INFO:teuthology.orchestra.console.smithi107:Power on
2024-04-14T16:10:19.877 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-04-14T16:10:19.885 INFO:teuthology.orchestra.console.smithi057:Power on
2024-04-14T16:10:19.886 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-04-14T16:10:19.892 INFO:teuthology.orchestra.console.smithi131:Power on
2024-04-14T16:10:19.892 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-04-14T16:10:19.899 INFO:teuthology.orchestra.console.smithi150:Power on
2024-04-14T16:10:19.899 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-04-14T16:10:19.906 DEBUG:teuthology.orchestra.console.smithi107:power on output: Chassis Power Control: Up/On
2024-04-14T16:10:19.906 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-14T16:10:19.913 DEBUG:teuthology.orchestra.console.smithi057:power on output: Chassis Power Control: Up/On
2024-04-14T16:10:19.913 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-14T16:10:19.920 DEBUG:teuthology.orchestra.console.smithi131:power on output: Chassis Power Control: Up/On
2024-04-14T16:10:19.920 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-14T16:10:19.926 DEBUG:teuthology.orchestra.console.smithi150:power on output: Chassis Power Control: Up/On
2024-04-14T16:10:19.926 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-14T16:10:19.933 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off
2024-04-14T16:10:19.933 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is off
2024-04-14T16:10:19.939 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is off
2024-04-14T16:10:19.946 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is off
2024-04-14T16:10:23.938 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-14T16:10:23.945 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-14T16:10:23.951 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-14T16:10:23.958 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-14T16:10:24.048 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on
2024-04-14T16:10:24.054 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is off
2024-04-14T16:10:24.060 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on
2024-04-14T16:10:24.066 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on
2024-04-14T16:10:24.149 INFO:teuthology.orchestra.console.smithi107:Power on completed
2024-04-14T16:10:24.160 INFO:teuthology.orchestra.console.smithi131:Power on completed
2024-04-14T16:10:24.168 INFO:teuthology.orchestra.console.smithi150:Power on completed
2024-04-14T16:10:24.250 INFO:teuthology.provision.fog.smithi107:Waiting for deploy to finish
2024-04-14T16:10:24.261 INFO:teuthology.provision.fog.smithi131:Waiting for deploy to finish
2024-04-14T16:10:24.269 INFO:teuthology.provision.fog.smithi150:Waiting for deploy to finish
2024-04-14T16:10:28.059 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-14T16:10:28.167 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on
2024-04-14T16:10:28.268 INFO:teuthology.orchestra.console.smithi057:Power on completed
2024-04-14T16:10:28.368 INFO:teuthology.provision.fog.smithi057:Waiting for deploy to finish
2024-04-14T16:12:55.091 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:13:10.158 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:13:10.161 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:13:13.214 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107
2024-04-14T16:13:13.630 WARNING:teuthology.provision.fog.smithi131:[Errno None] Unable to connect to port 22 on 172.21.15.131
2024-04-14T16:13:14.138 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:13:17.214 WARNING:teuthology.provision.fog.smithi057:[Errno None] Unable to connect to port 22 on 172.21.15.57
2024-04-14T16:13:19.217 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:13:19.632 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:13:22.270 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107
2024-04-14T16:13:22.686 WARNING:teuthology.provision.fog.smithi131:[Errno None] Unable to connect to port 22 on 172.21.15.131
2024-04-14T16:13:23.217 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:13:26.274 WARNING:teuthology.provision.fog.smithi057:[Errno None] Unable to connect to port 22 on 172.21.15.57
2024-04-14T16:13:28.273 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:13:28.688 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:13:31.326 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107
2024-04-14T16:13:31.742 WARNING:teuthology.provision.fog.smithi131:[Errno None] Unable to connect to port 22 on 172.21.15.131
2024-04-14T16:13:32.277 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:13:35.326 WARNING:teuthology.provision.fog.smithi057:[Errno None] Unable to connect to port 22 on 172.21.15.57
2024-04-14T16:13:37.329 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:13:37.744 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:13:40.382 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107
2024-04-14T16:13:40.798 WARNING:teuthology.provision.fog.smithi131:[Errno None] Unable to connect to port 22 on 172.21.15.131
2024-04-14T16:13:41.329 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:13:44.386 WARNING:teuthology.provision.fog.smithi057:[Errno None] Unable to connect to port 22 on 172.21.15.57
2024-04-14T16:13:46.384 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:13:46.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:13:49.442 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107
2024-04-14T16:13:50.389 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:13:53.470 WARNING:teuthology.provision.fog.smithi057:[Errno None] Unable to connect to port 22 on 172.21.15.57
2024-04-14T16:13:55.445 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:13:58.526 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107
2024-04-14T16:13:59.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:14:02.526 WARNING:teuthology.provision.fog.smithi057:[Errno None] Unable to connect to port 22 on 172.21.15.57
2024-04-14T16:14:04.530 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:14:05.347 WARNING:teuthology.provision.fog.smithi131:[Errno None] Unable to connect to port 22 on 172.21.15.131
2024-04-14T16:14:07.586 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107
2024-04-14T16:14:08.529 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:14:10.167 WARNING:teuthology.provision.fog.smithi150:timed out
2024-04-14T16:14:11.349 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:14:13.590 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:14:14.434 WARNING:teuthology.provision.fog.smithi131:[Errno None] Unable to connect to port 22 on 172.21.15.131
2024-04-14T16:14:16.169 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:14:20.439 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:14:23.518 WARNING:teuthology.provision.fog.smithi131:[Errno None] Unable to connect to port 22 on 172.21.15.131
2024-04-14T16:14:29.523 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:14:29.727 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-04-14T16:14:29.727 DEBUG:teuthology.orchestra.run.smithi131:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-04-14T16:14:50.047 INFO:teuthology.provision.fog.smithi131:Node is ready
2024-04-14T16:14:50.048 DEBUG:teuthology.orchestra.run.smithi131:> hostname
2024-04-14T16:14:50.067 INFO:teuthology.orchestra.run.smithi131.stdout:smithi131.front.sepia.ceph.com
2024-04-14T16:14:50.068 DEBUG:teuthology.orchestra.run.smithi131:> grep smithi131.front.sepia.ceph.com /etc/hosts
2024-04-14T16:14:50.125 INFO:teuthology.orchestra.run.smithi131.stdout:172.21.15.131 smithi131.front.sepia.ceph.com smithi131
2024-04-14T16:14:50.126 DEBUG:teuthology.orchestra.run.smithi131:> sudo hostname smithi131
2024-04-14T16:14:50.251 DEBUG:teuthology.orchestra.run.smithi131:> sudo sed -i -e 's/smithi131.front.sepia.ceph.com/smithi131/g' /etc/hosts
2024-04-14T16:14:50.282 DEBUG:teuthology.orchestra.run.smithi131:> sudo sed -i -e 's/172.21.15.131/172.21.15.131/g' /etc/hosts
2024-04-14T16:14:50.352 DEBUG:teuthology.orchestra.run.smithi131:> sudo sed -i -e 's/smithi131.front.sepia.ceph.com/smithi131/g' /etc/hostname
2024-04-14T16:14:50.422 DEBUG:teuthology.orchestra.run.smithi131:> sudo hostname smithi131
2024-04-14T16:14:50.490 DEBUG:teuthology.orchestra.run.smithi131:> cat /etc/os-release
2024-04-14T16:14:50.546 INFO:teuthology.orchestra.run.smithi131.stdout:NAME="CentOS Stream"
2024-04-14T16:14:50.546 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION="9"
2024-04-14T16:14:50.546 INFO:teuthology.orchestra.run.smithi131.stdout:ID="centos"
2024-04-14T16:14:50.546 INFO:teuthology.orchestra.run.smithi131.stdout:ID_LIKE="rhel fedora"
2024-04-14T16:14:50.546 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION_ID="9"
2024-04-14T16:14:50.546 INFO:teuthology.orchestra.run.smithi131.stdout:PLATFORM_ID="platform:el9"
2024-04-14T16:14:50.546 INFO:teuthology.orchestra.run.smithi131.stdout:PRETTY_NAME="CentOS Stream 9"
2024-04-14T16:14:50.546 INFO:teuthology.orchestra.run.smithi131.stdout:ANSI_COLOR="0;31"
2024-04-14T16:14:50.546 INFO:teuthology.orchestra.run.smithi131.stdout:LOGO="fedora-logo-icon"
2024-04-14T16:14:50.546 INFO:teuthology.orchestra.run.smithi131.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-04-14T16:14:50.546 INFO:teuthology.orchestra.run.smithi131.stdout:HOME_URL="https://centos.org/"
2024-04-14T16:14:50.547 INFO:teuthology.orchestra.run.smithi131.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-04-14T16:14:50.547 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-04-14T16:14:50.547 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-04-14T16:14:50.547 INFO:teuthology.provision.fog.smithi131:Deploy complete!
2024-04-14T16:14:50.549 DEBUG:teuthology.parallel:result is None
2024-04-14T16:15:08.547 WARNING:teuthology.provision.fog.smithi057:timed out
2024-04-14T16:15:13.594 WARNING:teuthology.provision.fog.smithi107:timed out
2024-04-14T16:15:14.550 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:15:14.811 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-04-14T16:15:14.812 DEBUG:teuthology.orchestra.run.smithi057:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-04-14T16:15:15.148 INFO:teuthology.provision.fog.smithi057:Node is ready
2024-04-14T16:15:15.148 DEBUG:teuthology.orchestra.run.smithi057:> hostname
2024-04-14T16:15:15.164 INFO:teuthology.orchestra.run.smithi057.stdout:smithi057.front.sepia.ceph.com
2024-04-14T16:15:15.164 DEBUG:teuthology.orchestra.run.smithi057:> grep smithi057.front.sepia.ceph.com /etc/hosts
2024-04-14T16:15:15.220 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57 smithi057.front.sepia.ceph.com smithi057
2024-04-14T16:15:15.220 DEBUG:teuthology.orchestra.run.smithi057:> sudo hostname smithi057
2024-04-14T16:15:15.319 DEBUG:teuthology.orchestra.run.smithi057:> sudo sed -i -e 's/smithi057.front.sepia.ceph.com/smithi057/g' /etc/hosts
2024-04-14T16:15:15.350 DEBUG:teuthology.orchestra.run.smithi057:> sudo sed -i -e 's/172.21.15.57/172.21.15.57/g' /etc/hosts
2024-04-14T16:15:15.420 DEBUG:teuthology.orchestra.run.smithi057:> sudo sed -i -e 's/smithi057.front.sepia.ceph.com/smithi057/g' /etc/hostname
2024-04-14T16:15:15.491 DEBUG:teuthology.orchestra.run.smithi057:> sudo hostname smithi057
2024-04-14T16:15:15.562 DEBUG:teuthology.orchestra.run.smithi057:> cat /etc/os-release
2024-04-14T16:15:15.616 INFO:teuthology.orchestra.run.smithi057.stdout:NAME="CentOS Stream"
2024-04-14T16:15:15.617 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION="9"
2024-04-14T16:15:15.617 INFO:teuthology.orchestra.run.smithi057.stdout:ID="centos"
2024-04-14T16:15:15.617 INFO:teuthology.orchestra.run.smithi057.stdout:ID_LIKE="rhel fedora"
2024-04-14T16:15:15.617 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION_ID="9"
2024-04-14T16:15:15.617 INFO:teuthology.orchestra.run.smithi057.stdout:PLATFORM_ID="platform:el9"
2024-04-14T16:15:15.617 INFO:teuthology.orchestra.run.smithi057.stdout:PRETTY_NAME="CentOS Stream 9"
2024-04-14T16:15:15.617 INFO:teuthology.orchestra.run.smithi057.stdout:ANSI_COLOR="0;31"
2024-04-14T16:15:15.617 INFO:teuthology.orchestra.run.smithi057.stdout:LOGO="fedora-logo-icon"
2024-04-14T16:15:15.617 INFO:teuthology.orchestra.run.smithi057.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-04-14T16:15:15.617 INFO:teuthology.orchestra.run.smithi057.stdout:HOME_URL="https://centos.org/"
2024-04-14T16:15:15.617 INFO:teuthology.orchestra.run.smithi057.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-04-14T16:15:15.617 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-04-14T16:15:15.618 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-04-14T16:15:15.618 INFO:teuthology.provision.fog.smithi057:Deploy complete!
2024-04-14T16:15:15.618 DEBUG:teuthology.parallel:result is None
2024-04-14T16:15:16.172 WARNING:teuthology.provision.fog.smithi150:timed out
2024-04-14T16:15:19.599 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:15:19.861 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-04-14T16:15:19.861 DEBUG:teuthology.orchestra.run.smithi107:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-04-14T16:15:20.169 INFO:teuthology.provision.fog.smithi107:Node is ready
2024-04-14T16:15:20.169 DEBUG:teuthology.orchestra.run.smithi107:> hostname
2024-04-14T16:15:20.184 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107.front.sepia.ceph.com
2024-04-14T16:15:20.185 DEBUG:teuthology.orchestra.run.smithi107:> grep smithi107.front.sepia.ceph.com /etc/hosts
2024-04-14T16:15:20.240 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107 smithi107.front.sepia.ceph.com smithi107
2024-04-14T16:15:20.240 DEBUG:teuthology.orchestra.run.smithi107:> sudo hostname smithi107
2024-04-14T16:15:20.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i -e 's/smithi107.front.sepia.ceph.com/smithi107/g' /etc/hosts
2024-04-14T16:15:20.378 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i -e 's/172.21.15.107/172.21.15.107/g' /etc/hosts
2024-04-14T16:15:20.447 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i -e 's/smithi107.front.sepia.ceph.com/smithi107/g' /etc/hostname
2024-04-14T16:15:20.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo hostname smithi107
2024-04-14T16:15:20.582 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release
2024-04-14T16:15:20.637 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Stream"
2024-04-14T16:15:20.637 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="9"
2024-04-14T16:15:20.637 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos"
2024-04-14T16:15:20.637 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora"
2024-04-14T16:15:20.637 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="9"
2024-04-14T16:15:20.637 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el9"
2024-04-14T16:15:20.637 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Stream 9"
2024-04-14T16:15:20.637 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31"
2024-04-14T16:15:20.637 INFO:teuthology.orchestra.run.smithi107.stdout:LOGO="fedora-logo-icon"
2024-04-14T16:15:20.637 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-04-14T16:15:20.638 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://centos.org/"
2024-04-14T16:15:20.638 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-04-14T16:15:20.638 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-04-14T16:15:20.638 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-04-14T16:15:20.638 INFO:teuthology.provision.fog.smithi107:Deploy complete!
2024-04-14T16:15:20.639 DEBUG:teuthology.parallel:result is None
2024-04-14T16:15:22.175 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:15:22.176 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150
2024-04-14T16:15:28.183 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:16:28.243 WARNING:teuthology.provision.fog.smithi150:timed out
2024-04-14T16:16:34.247 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:16:37.311 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150
2024-04-14T16:16:43.315 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:16:46.366 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150
2024-04-14T16:16:52.374 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:16:55.458 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150
2024-04-14T16:17:01.465 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:17:19.903 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150
2024-04-14T16:17:25.907 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:17:28.958 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150
2024-04-14T16:17:34.963 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:17:35.102 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-04-14T16:17:39.237 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-04-14T16:17:46.333 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-04-14T16:17:56.432 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-04-14T16:18:09.569 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-04-14T16:18:25.707 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-04-14T16:19:03.071 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150
2024-04-14T16:19:09.073 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:19:11.262 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150
2024-04-14T16:19:17.264 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:19:20.318 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150
2024-04-14T16:19:26.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:19:29.374 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150
2024-04-14T16:19:35.376 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:19:53.758 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150
2024-04-14T16:19:59.760 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:20:02.814 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150
2024-04-14T16:20:08.816 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:20:11.870 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150
2024-04-14T16:20:17.872 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-14T16:20:18.138 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-04-14T16:20:18.138 DEBUG:teuthology.orchestra.run.smithi150:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-04-14T16:20:38.507 INFO:teuthology.provision.fog.smithi150:Node is ready
2024-04-14T16:20:38.507 DEBUG:teuthology.orchestra.run.smithi150:> hostname
2024-04-14T16:20:38.525 INFO:teuthology.orchestra.run.smithi150.stdout:smithi150.front.sepia.ceph.com
2024-04-14T16:20:38.526 DEBUG:teuthology.orchestra.run.smithi150:> grep smithi150.front.sepia.ceph.com /etc/hosts
2024-04-14T16:20:38.582 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150 smithi150.front.sepia.ceph.com smithi150
2024-04-14T16:20:38.583 DEBUG:teuthology.orchestra.run.smithi150:> sudo hostname smithi150
2024-04-14T16:20:38.723 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i -e 's/smithi150.front.sepia.ceph.com/smithi150/g' /etc/hosts
2024-04-14T16:20:38.761 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i -e 's/172.21.15.150/172.21.15.150/g' /etc/hosts
2024-04-14T16:20:38.830 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i -e 's/smithi150.front.sepia.ceph.com/smithi150/g' /etc/hostname
2024-04-14T16:20:38.901 DEBUG:teuthology.orchestra.run.smithi150:> sudo hostname smithi150
2024-04-14T16:20:38.968 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/os-release
2024-04-14T16:20:39.022 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="CentOS Stream"
2024-04-14T16:20:39.022 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="9"
2024-04-14T16:20:39.022 INFO:teuthology.orchestra.run.smithi150.stdout:ID="centos"
2024-04-14T16:20:39.022 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="rhel fedora"
2024-04-14T16:20:39.022 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="9"
2024-04-14T16:20:39.022 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el9"
2024-04-14T16:20:39.022 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="CentOS Stream 9"
2024-04-14T16:20:39.022 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31"
2024-04-14T16:20:39.022 INFO:teuthology.orchestra.run.smithi150.stdout:LOGO="fedora-logo-icon"
2024-04-14T16:20:39.022 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-04-14T16:20:39.022 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://centos.org/"
2024-04-14T16:20:39.022 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-04-14T16:20:39.023 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-04-14T16:20:39.023 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-04-14T16:20:39.023 INFO:teuthology.provision.fog.smithi150:Deploy complete!
2024-04-14T16:20:39.023 DEBUG:teuthology.parallel:result is None
2024-04-14T16:20:40.203 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/lock/ops.py", line 326, in reimage_machines
    reimaged = do_update_keys(list(reimaged.keys()))[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/lock/ops.py", line 290, in do_update_keys
    keys_dict = misc.ssh_keyscan(machines, _raise=_raise)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/misc.py", line 1108, in ssh_keyscan
    key = _ssh_keyscan(hostname)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/misc.py", line 1145, in _ssh_keyscan
    return sorted(keys)[0]
IndexError: list index out of range
2024-04-14T16:20:40.205 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-04-14T16:20:40.490 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=b9a1da77b66f494daf247cf7d0cd7886
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/lock/ops.py", line 326, in reimage_machines
    reimaged = do_update_keys(list(reimaged.keys()))[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/lock/ops.py", line 290, in do_update_keys
    keys_dict = misc.ssh_keyscan(machines, _raise=_raise)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/misc.py", line 1108, in ssh_keyscan
    key = _ssh_keyscan(hostname)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/misc.py", line 1145, in _ssh_keyscan
    return sorted(keys)[0]
IndexError: list index out of range
2024-04-14T16:20:40.492 INFO:teuthology.nuke:Checking targets against current locks
2024-04-14T16:20:40.553 DEBUG:teuthology.nuke:shortname: smithi057
2024-04-14T16:20:40.553 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-04-14T16:20:40.583 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi057.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-09_20:16:14-rbd-main-distro-default-smithi/7649443', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-14 16:10:17.872110', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCQCS6hi1mpPwpjOAPYCI7uq9wgPORJK3TldH4Sno8vA/j9/8tF8ZWHd1K8icA6/S7iqkum48UZgf1kWJyuKkGY='}
2024-04-14T16:20:40.584 DEBUG:teuthology.nuke:shortname: smithi107
2024-04-14T16:20:40.584 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-04-14T16:20:40.611 DEBUG:teuthology.nuke:shortname: smithi131
2024-04-14T16:20:40.611 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-04-14T16:20:40.615 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-09_20:16:14-rbd-main-distro-default-smithi/7649443', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-14 16:10:17.871168', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHuUS6rtV+F6l3e0dgTgTkL/+ImZZgdQvBCuAZXfHmkN5NmyAPPDzMz1azTauBH04/y/FTYrtW5/Y0G8M6SCnQk='}
2024-04-14T16:20:40.641 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi131.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-09_20:16:14-rbd-main-distro-default-smithi/7649448', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-14 16:15:40.542834', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGWwe4ekX1RrZe43+TfBddCmm/3WhBrUeveWWJ3gqC4LB3SDDhv4qgESODsxVYUX2ts63HAh942fvSb15Iv3ET0='}
2024-04-14T16:20:40.645 DEBUG:teuthology.nuke:shortname: smithi150
2024-04-14T16:20:40.645 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-04-14T16:20:40.652 INFO:teuthology.orchestra.console.smithi057:Power off
2024-04-14T16:20:40.652 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-14T16:20:40.669 INFO:teuthology.orchestra.console.smithi107:Power off
2024-04-14T16:20:40.670 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-14T16:20:40.686 DEBUG:teuthology.orchestra.console.smithi057:power off output: Chassis Power Control: Down/Off
2024-04-14T16:20:40.687 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-14T16:20:40.696 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-09_20:16:14-rbd-main-distro-default-smithi/7649448', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-14 16:15:40.541538', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMqwswwB32gHnh9bdN2pCrT87T4FqHcjSi9LTvsGPJrD6qr/D9BvBeO8hJMIOrHx+0eBPJzO9kc6rWC24HHIDHY='}
2024-04-14T16:20:40.699 DEBUG:teuthology.orchestra.console.smithi107:power off output: Chassis Power Control: Down/Off
2024-04-14T16:20:40.699 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-14T16:20:40.712 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on
2024-04-14T16:20:40.717 INFO:teuthology.orchestra.console.smithi131:Power off
2024-04-14T16:20:40.717 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-14T16:20:40.728 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on
2024-04-14T16:20:40.742 DEBUG:teuthology.orchestra.console.smithi131:power off output: Chassis Power Control: Down/Off
2024-04-14T16:20:40.742 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-14T16:20:40.759 INFO:teuthology.orchestra.console.smithi150:Power off
2024-04-14T16:20:40.760 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-14T16:20:40.769 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on
2024-04-14T16:20:40.782 DEBUG:teuthology.orchestra.console.smithi150:power off output: Chassis Power Control: Down/Off
2024-04-14T16:20:40.782 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-14T16:20:40.804 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on
2024-04-14T16:20:44.714 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-14T16:20:44.729 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-14T16:20:44.770 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-14T16:20:44.805 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-14T16:20:44.823 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on
2024-04-14T16:20:44.838 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on
2024-04-14T16:20:44.879 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on
2024-04-14T16:20:44.912 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on
2024-04-14T16:20:48.823 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-14T16:20:48.839 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-14T16:20:48.880 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-14T16:20:48.913 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-14T16:20:48.934 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is off
2024-04-14T16:20:48.948 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on
2024-04-14T16:20:48.991 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is off
2024-04-14T16:20:49.024 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is off
2024-04-14T16:20:49.035 INFO:teuthology.orchestra.console.smithi057:Power off completed
2024-04-14T16:20:49.092 INFO:teuthology.orchestra.console.smithi131:Power off completed
2024-04-14T16:20:49.125 INFO:teuthology.orchestra.console.smithi150:Power off completed
2024-04-14T16:20:49.198 INFO:teuthology.lock.ops:unlocked: smithi057.front.sepia.ceph.com
2024-04-14T16:20:49.237 INFO:teuthology.lock.ops:unlocked: smithi131.front.sepia.ceph.com
2024-04-14T16:20:49.271 INFO:teuthology.lock.ops:unlocked: smithi150.front.sepia.ceph.com
2024-04-14T16:20:52.949 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-14T16:20:53.059 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off
2024-04-14T16:20:53.160 INFO:teuthology.orchestra.console.smithi107:Power off completed
2024-04-14T16:20:53.304 INFO:teuthology.lock.ops:unlocked: smithi107.front.sepia.ceph.com
2024-04-14T16:20:53.810 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/lock/ops.py", line 326, in reimage_machines
    reimaged = do_update_keys(list(reimaged.keys()))[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/lock/ops.py", line 290, in do_update_keys
    keys_dict = misc.ssh_keyscan(machines, _raise=_raise)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/misc.py", line 1108, in ssh_keyscan
    key = _ssh_keyscan(hostname)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/misc.py", line 1145, in _ssh_keyscan
    return sorted(keys)[0]
IndexError: list index out of range