2024-03-20T05:00:55.386 INFO:root:teuthology version: 0.0.1.dev262+ge691533
2024-03-20T05:00:55.387 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-20T05:00:55.584 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f41afeeaf40>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f41b0f4bc10>, signals=[15])
2024-03-20T05:00:55.585 INFO:teuthology.lock.ops:Start node 'smithi078.front.sepia.ceph.com' reimaging
2024-03-20T05:00:55.585 INFO:teuthology.lock.ops:Updating [smithi078.front.sepia.ceph.com]: reset os type and version on server
2024-03-20T05:00:55.585 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server
2024-03-20T05:00:55.607 INFO:teuthology.lock.ops:Start node 'smithi100.front.sepia.ceph.com' reimaging
2024-03-20T05:00:55.607 INFO:teuthology.lock.ops:Updating [smithi100.front.sepia.ceph.com]: reset os type and version on server
2024-03-20T05:00:55.608 INFO:teuthology.lock.ops:Updating smithi100.front.sepia.ceph.com on lock server
2024-03-20T05:00:55.624 INFO:teuthology.lock.ops:Start node 'smithi102.front.sepia.ceph.com' reimaging
2024-03-20T05:00:55.624 INFO:teuthology.lock.ops:Updating [smithi102.front.sepia.ceph.com]: reset os type and version on server
2024-03-20T05:00:55.624 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server
2024-03-20T05:00:55.641 INFO:teuthology.lock.ops:Start node 'smithi191.front.sepia.ceph.com' reimaging
2024-03-20T05:00:55.641 INFO:teuthology.lock.ops:Updating [smithi191.front.sepia.ceph.com]: reset os type and version on server
2024-03-20T05:00:55.641 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server
2024-03-20T05:00:55.713 DEBUG:teuthology.provision.fog.smithi078:Requesting image smithi_centos_9.stream (ID 10)
2024-03-20T05:00:55.728 DEBUG:teuthology.provision.fog.smithi100:Requesting image smithi_centos_9.stream (ID 10)
2024-03-20T05:00:55.737 DEBUG:teuthology.provision.fog.smithi102:Requesting image smithi_centos_9.stream (ID 10)
2024-03-20T05:00:55.766 DEBUG:teuthology.provision.fog.smithi191:Requesting image smithi_centos_9.stream (ID 10)
2024-03-20T05:00:55.770 INFO:teuthology.provision.fog.smithi078:Scheduling deploy of centos 9.stream
2024-03-20T05:00:55.770 INFO:teuthology.provision.fog.smithi100:Scheduling deploy of centos 9.stream
2024-03-20T05:00:55.778 INFO:teuthology.provision.fog.smithi102:Scheduling deploy of centos 9.stream
2024-03-20T05:00:55.811 INFO:teuthology.provision.fog.smithi191:Scheduling deploy of centos 9.stream
2024-03-20T05:00:55.861 DEBUG:teuthology.provision.fog.smithi102:Canceling deploy task with ID 875738
2024-03-20T05:00:55.875 DEBUG:teuthology.provision.fog.smithi100:Canceling deploy task with ID 875739
2024-03-20T05:00:55.905 DEBUG:teuthology.provision.fog.smithi191:Canceling deploy task with ID 875740
2024-03-20T05:00:56.112 INFO:teuthology.orchestra.console.smithi102:Power off
2024-03-20T05:00:56.112 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-20T05:00:56.126 INFO:teuthology.orchestra.console.smithi078:Power off
2024-03-20T05:00:56.126 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-20T05:00:56.133 DEBUG:teuthology.orchestra.console.smithi102:power off output: Chassis Power Control: Down/Off
2024-03-20T05:00:56.134 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:00:56.158 DEBUG:teuthology.orchestra.console.smithi078:power off output: Chassis Power Control: Down/Off
2024-03-20T05:00:56.158 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:00:56.167 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is off
2024-03-20T05:00:56.179 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is off
2024-03-20T05:00:56.190 INFO:teuthology.orchestra.console.smithi191:Power off
2024-03-20T05:00:56.191 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-20T05:00:56.203 INFO:teuthology.orchestra.console.smithi100:Power off
2024-03-20T05:00:56.204 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-20T05:00:56.212 DEBUG:teuthology.orchestra.console.smithi191:power off output: Chassis Power Control: Down/Off
2024-03-20T05:00:56.213 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:00:56.224 DEBUG:teuthology.orchestra.console.smithi100:power off output: Chassis Power Control: Down/Off
2024-03-20T05:00:56.224 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:00:56.233 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is off
2024-03-20T05:00:56.245 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is off
2024-03-20T05:00:56.269 INFO:teuthology.orchestra.console.smithi102:Power off completed
2024-03-20T05:00:56.280 INFO:teuthology.orchestra.console.smithi078:Power off completed
2024-03-20T05:00:56.335 INFO:teuthology.orchestra.console.smithi191:Power off completed
2024-03-20T05:00:56.346 INFO:teuthology.orchestra.console.smithi100:Power off completed
2024-03-20T05:00:56.371 INFO:teuthology.orchestra.console.smithi102:Power on
2024-03-20T05:00:56.371 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-03-20T05:00:56.381 INFO:teuthology.orchestra.console.smithi078:Power on
2024-03-20T05:00:56.381 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-03-20T05:00:56.392 DEBUG:teuthology.orchestra.console.smithi102:power on output: Chassis Power Control: Up/On
2024-03-20T05:00:56.393 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:00:56.405 DEBUG:teuthology.orchestra.console.smithi078:power on output: Chassis Power Control: Up/On
2024-03-20T05:00:56.407 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:00:56.415 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is off
2024-03-20T05:00:56.428 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is off
2024-03-20T05:00:56.435 INFO:teuthology.orchestra.console.smithi191:Power on
2024-03-20T05:00:56.436 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-03-20T05:00:56.447 INFO:teuthology.orchestra.console.smithi100:Power on
2024-03-20T05:00:56.448 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-03-20T05:00:56.456 DEBUG:teuthology.orchestra.console.smithi191:power on output: Chassis Power Control: Up/On
2024-03-20T05:00:56.456 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:00:56.469 DEBUG:teuthology.orchestra.console.smithi100:power on output: Chassis Power Control: Up/On
2024-03-20T05:00:56.469 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:00:56.477 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is off
2024-03-20T05:00:56.489 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is off
2024-03-20T05:01:00.418 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:01:00.428 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:01:00.478 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:01:00.489 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:01:00.528 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on
2024-03-20T05:01:00.537 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is off
2024-03-20T05:01:00.588 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is off
2024-03-20T05:01:00.598 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is on
2024-03-20T05:01:00.630 INFO:teuthology.orchestra.console.smithi102:Power on completed
2024-03-20T05:01:00.700 INFO:teuthology.orchestra.console.smithi100:Power on completed
2024-03-20T05:01:00.731 INFO:teuthology.provision.fog.smithi102:Waiting for deploy to finish
2024-03-20T05:01:00.800 INFO:teuthology.provision.fog.smithi100:Waiting for deploy to finish
2024-03-20T05:01:04.542 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:01:04.590 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:01:04.651 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on
2024-03-20T05:01:04.752 INFO:teuthology.orchestra.console.smithi078:Power on completed
2024-03-20T05:01:04.767 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on
2024-03-20T05:01:04.854 INFO:teuthology.provision.fog.smithi078:Waiting for deploy to finish
2024-03-20T05:01:04.868 INFO:teuthology.orchestra.console.smithi191:Power on completed
2024-03-20T05:01:04.969 INFO:teuthology.provision.fog.smithi191:Waiting for deploy to finish
2024-03-20T05:03:35.891 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-20T05:03:36.190 ERROR:teuthology.orchestra.connection:Error authenticating with smithi191.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-03-20T05:03:46.847 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-20T05:03:46.848 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-20T05:03:50.950 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-20T05:03:54.014 WARNING:teuthology.provision.fog.smithi078:[Errno None] Unable to connect to port 22 on 172.21.15.78
2024-03-20T05:04:00.301 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-20T05:04:03.358 WARNING:teuthology.provision.fog.smithi078:[Errno None] Unable to connect to port 22 on 172.21.15.78
2024-03-20T05:04:09.360 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-20T05:04:12.414 WARNING:teuthology.provision.fog.smithi078:[Errno None] Unable to connect to port 22 on 172.21.15.78
2024-03-20T05:04:18.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-20T05:04:21.474 WARNING:teuthology.provision.fog.smithi078:[Errno None] Unable to connect to port 22 on 172.21.15.78
2024-03-20T05:04:21.534 WARNING:teuthology.provision.fog.smithi102:[Errno None] Unable to connect to port 22 on 172.21.15.102
2024-03-20T05:04:21.535 WARNING:teuthology.provision.fog.smithi100:[Errno None] Unable to connect to port 22 on 172.21.15.100
2024-03-20T05:04:27.475 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-20T05:04:27.535 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-20T05:04:27.537 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-20T05:04:28.706 WARNING:teuthology.provision.fog.smithi102:[Errno None] Unable to connect to port 22 on 172.21.15.102
2024-03-20T05:04:30.526 WARNING:teuthology.provision.fog.smithi078:[Errno None] Unable to connect to port 22 on 172.21.15.78
2024-03-20T05:04:30.594 WARNING:teuthology.provision.fog.smithi100:[Errno None] Unable to connect to port 22 on 172.21.15.100
2024-03-20T05:04:34.708 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-20T05:04:36.528 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-20T05:04:36.595 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-20T05:04:36.734 WARNING:teuthology.provision.fog.smithi100:[Errno None] Unable to connect to port 22 on 172.21.15.100
2024-03-20T05:04:37.758 WARNING:teuthology.provision.fog.smithi102:[Errno None] Unable to connect to port 22 on 172.21.15.102
2024-03-20T05:04:39.582 WARNING:teuthology.provision.fog.smithi078:[Errno None] Unable to connect to port 22 on 172.21.15.78
2024-03-20T05:04:40.193 WARNING:teuthology.provision.fog.smithi191:timed out
2024-03-20T05:04:42.736 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-20T05:04:43.760 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-20T05:04:45.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-20T05:04:46.194 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-20T05:04:48.638 WARNING:teuthology.provision.fog.smithi078:[Errno None] Unable to connect to port 22 on 172.21.15.78
2024-03-20T05:04:53.022 WARNING:teuthology.provision.fog.smithi100:[Errno None] Unable to connect to port 22 on 172.21.15.100
2024-03-20T05:04:54.640 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-20T05:04:59.023 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-20T05:05:02.078 WARNING:teuthology.provision.fog.smithi100:[Errno None] Unable to connect to port 22 on 172.21.15.100
2024-03-20T05:05:02.238 WARNING:teuthology.provision.fog.smithi102:[Errno None] Unable to connect to port 22 on 172.21.15.102
2024-03-20T05:05:08.080 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-20T05:05:08.222 WARNING:teuthology.provision.fog.smithi100:[Errno None] Unable to connect to port 22 on 172.21.15.100
2024-03-20T05:05:08.240 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-20T05:05:11.294 WARNING:teuthology.provision.fog.smithi102:[Errno None] Unable to connect to port 22 on 172.21.15.102
2024-03-20T05:05:14.224 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-20T05:05:17.278 WARNING:teuthology.provision.fog.smithi100:[Errno None] Unable to connect to port 22 on 172.21.15.100
2024-03-20T05:05:17.296 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-20T05:05:17.859 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-03-20T05:05:17.860 DEBUG:teuthology.orchestra.run.smithi191:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-03-20T05:05:20.350 WARNING:teuthology.provision.fog.smithi102:[Errno None] Unable to connect to port 22 on 172.21.15.102
2024-03-20T05:05:23.280 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-20T05:05:23.367 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-03-20T05:05:23.368 DEBUG:teuthology.orchestra.run.smithi100:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-03-20T05:05:26.353 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-20T05:05:26.689 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-03-20T05:05:26.689 DEBUG:teuthology.orchestra.run.smithi102:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-03-20T05:05:38.189 INFO:teuthology.provision.fog.smithi191:Node is ready
2024-03-20T05:05:38.189 DEBUG:teuthology.orchestra.run.smithi191:> hostname
2024-03-20T05:05:38.207 INFO:teuthology.orchestra.run.smithi191.stdout:smithi191.front.sepia.ceph.com
2024-03-20T05:05:38.207 DEBUG:teuthology.orchestra.run.smithi191:> grep smithi191.front.sepia.ceph.com /etc/hosts
2024-03-20T05:05:38.265 INFO:teuthology.orchestra.run.smithi191.stdout:172.21.15.191 smithi191.front.sepia.ceph.com smithi191
2024-03-20T05:05:38.265 DEBUG:teuthology.orchestra.run.smithi191:> sudo hostname smithi191
2024-03-20T05:05:38.399 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i -e 's/smithi191.front.sepia.ceph.com/smithi191/g' /etc/hosts
2024-03-20T05:05:38.429 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i -e 's/172.21.15.191/172.21.15.191/g' /etc/hosts
2024-03-20T05:05:38.454 INFO:teuthology.provision.fog.smithi100:Node is ready
2024-03-20T05:05:38.454 DEBUG:teuthology.orchestra.run.smithi100:> hostname
2024-03-20T05:05:38.471 INFO:teuthology.orchestra.run.smithi100.stdout:smithi100.front.sepia.ceph.com
2024-03-20T05:05:38.472 DEBUG:teuthology.orchestra.run.smithi100:> grep smithi100.front.sepia.ceph.com /etc/hosts
2024-03-20T05:05:38.498 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i -e 's/smithi191.front.sepia.ceph.com/smithi191/g' /etc/hostname
2024-03-20T05:05:38.544 INFO:teuthology.orchestra.run.smithi100.stdout:172.21.15.100 smithi100.front.sepia.ceph.com smithi100
2024-03-20T05:05:38.545 DEBUG:teuthology.orchestra.run.smithi100:> sudo hostname smithi100
2024-03-20T05:05:38.567 DEBUG:teuthology.orchestra.run.smithi191:> sudo hostname smithi191
2024-03-20T05:05:38.635 DEBUG:teuthology.orchestra.run.smithi191:> cat /etc/os-release
2024-03-20T05:05:38.665 DEBUG:teuthology.orchestra.run.smithi100:> sudo sed -i -e 's/smithi100.front.sepia.ceph.com/smithi100/g' /etc/hosts
2024-03-20T05:05:38.690 INFO:teuthology.orchestra.run.smithi191.stdout:NAME="CentOS Stream"
2024-03-20T05:05:38.690 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION="9"
2024-03-20T05:05:38.690 INFO:teuthology.orchestra.run.smithi191.stdout:ID="centos"
2024-03-20T05:05:38.690 INFO:teuthology.orchestra.run.smithi191.stdout:ID_LIKE="rhel fedora"
2024-03-20T05:05:38.691 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_ID="9"
2024-03-20T05:05:38.691 INFO:teuthology.orchestra.run.smithi191.stdout:PLATFORM_ID="platform:el9"
2024-03-20T05:05:38.691 INFO:teuthology.orchestra.run.smithi191.stdout:PRETTY_NAME="CentOS Stream 9"
2024-03-20T05:05:38.691 INFO:teuthology.orchestra.run.smithi191.stdout:ANSI_COLOR="0;31"
2024-03-20T05:05:38.691 INFO:teuthology.orchestra.run.smithi191.stdout:LOGO="fedora-logo-icon"
2024-03-20T05:05:38.691 INFO:teuthology.orchestra.run.smithi191.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-03-20T05:05:38.691 INFO:teuthology.orchestra.run.smithi191.stdout:HOME_URL="https://centos.org/"
2024-03-20T05:05:38.691 INFO:teuthology.orchestra.run.smithi191.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-03-20T05:05:38.691 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-03-20T05:05:38.691 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-03-20T05:05:38.692 INFO:teuthology.provision.fog.smithi191:Deploy complete!
2024-03-20T05:05:38.693 DEBUG:teuthology.parallel:result is None
2024-03-20T05:05:38.694 DEBUG:teuthology.orchestra.run.smithi100:> sudo sed -i -e 's/172.21.15.100/172.21.15.100/g' /etc/hosts
2024-03-20T05:05:38.762 DEBUG:teuthology.orchestra.run.smithi100:> sudo sed -i -e 's/smithi100.front.sepia.ceph.com/smithi100/g' /etc/hostname
2024-03-20T05:05:38.829 DEBUG:teuthology.orchestra.run.smithi100:> sudo hostname smithi100
2024-03-20T05:05:38.896 DEBUG:teuthology.orchestra.run.smithi100:> cat /etc/os-release
2024-03-20T05:05:38.950 INFO:teuthology.orchestra.run.smithi100.stdout:NAME="CentOS Stream"
2024-03-20T05:05:38.950 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION="9"
2024-03-20T05:05:38.950 INFO:teuthology.orchestra.run.smithi100.stdout:ID="centos"
2024-03-20T05:05:38.950 INFO:teuthology.orchestra.run.smithi100.stdout:ID_LIKE="rhel fedora"
2024-03-20T05:05:38.950 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION_ID="9"
2024-03-20T05:05:38.950 INFO:teuthology.orchestra.run.smithi100.stdout:PLATFORM_ID="platform:el9"
2024-03-20T05:05:38.950 INFO:teuthology.orchestra.run.smithi100.stdout:PRETTY_NAME="CentOS Stream 9"
2024-03-20T05:05:38.950 INFO:teuthology.orchestra.run.smithi100.stdout:ANSI_COLOR="0;31"
2024-03-20T05:05:38.950 INFO:teuthology.orchestra.run.smithi100.stdout:LOGO="fedora-logo-icon"
2024-03-20T05:05:38.951 INFO:teuthology.orchestra.run.smithi100.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-03-20T05:05:38.951 INFO:teuthology.orchestra.run.smithi100.stdout:HOME_URL="https://centos.org/"
2024-03-20T05:05:38.951 INFO:teuthology.orchestra.run.smithi100.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-03-20T05:05:38.951 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-03-20T05:05:38.951 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-03-20T05:05:38.951 INFO:teuthology.provision.fog.smithi100:Deploy complete!
2024-03-20T05:05:38.951 DEBUG:teuthology.parallel:result is None
2024-03-20T05:05:42.150 INFO:teuthology.provision.fog.smithi102:Node is ready
2024-03-20T05:05:42.150 DEBUG:teuthology.orchestra.run.smithi102:> hostname
2024-03-20T05:05:42.173 INFO:teuthology.orchestra.run.smithi102.stdout:smithi102.front.sepia.ceph.com
2024-03-20T05:05:42.173 DEBUG:teuthology.orchestra.run.smithi102:> grep smithi102.front.sepia.ceph.com /etc/hosts
2024-03-20T05:05:42.230 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102 smithi102.front.sepia.ceph.com smithi102
2024-03-20T05:05:42.230 DEBUG:teuthology.orchestra.run.smithi102:> sudo hostname smithi102
2024-03-20T05:05:42.363 DEBUG:teuthology.orchestra.run.smithi102:> sudo sed -i -e 's/smithi102.front.sepia.ceph.com/smithi102/g' /etc/hosts
2024-03-20T05:05:42.397 DEBUG:teuthology.orchestra.run.smithi102:> sudo sed -i -e 's/172.21.15.102/172.21.15.102/g' /etc/hosts
2024-03-20T05:05:42.465 DEBUG:teuthology.orchestra.run.smithi102:> sudo sed -i -e 's/smithi102.front.sepia.ceph.com/smithi102/g' /etc/hostname
2024-03-20T05:05:42.533 DEBUG:teuthology.orchestra.run.smithi102:> sudo hostname smithi102
2024-03-20T05:05:42.599 DEBUG:teuthology.orchestra.run.smithi102:> cat /etc/os-release
2024-03-20T05:05:42.653 INFO:teuthology.orchestra.run.smithi102.stdout:NAME="CentOS Stream"
2024-03-20T05:05:42.654 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION="9"
2024-03-20T05:05:42.654 INFO:teuthology.orchestra.run.smithi102.stdout:ID="centos"
2024-03-20T05:05:42.654 INFO:teuthology.orchestra.run.smithi102.stdout:ID_LIKE="rhel fedora"
2024-03-20T05:05:42.654 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION_ID="9"
2024-03-20T05:05:42.654 INFO:teuthology.orchestra.run.smithi102.stdout:PLATFORM_ID="platform:el9"
2024-03-20T05:05:42.654 INFO:teuthology.orchestra.run.smithi102.stdout:PRETTY_NAME="CentOS Stream 9"
2024-03-20T05:05:42.654 INFO:teuthology.orchestra.run.smithi102.stdout:ANSI_COLOR="0;31"
2024-03-20T05:05:42.654 INFO:teuthology.orchestra.run.smithi102.stdout:LOGO="fedora-logo-icon"
2024-03-20T05:05:42.654 INFO:teuthology.orchestra.run.smithi102.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-03-20T05:05:42.654 INFO:teuthology.orchestra.run.smithi102.stdout:HOME_URL="https://centos.org/"
2024-03-20T05:05:42.655 INFO:teuthology.orchestra.run.smithi102.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-03-20T05:05:42.655 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-03-20T05:05:42.655 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-03-20T05:05:42.655 INFO:teuthology.provision.fog.smithi102:Deploy complete!
2024-03-20T05:05:42.655 DEBUG:teuthology.parallel:result is None
2024-03-20T05:05:54.642 WARNING:teuthology.provision.fog.smithi078:timed out
2024-03-20T05:06:00.644 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-20T05:06:00.949 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-03-20T05:06:00.950 DEBUG:teuthology.orchestra.run.smithi078:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-03-20T05:06:01.282 INFO:teuthology.provision.fog.smithi078:Node is ready
2024-03-20T05:06:01.283 DEBUG:teuthology.orchestra.run.smithi078:> hostname
2024-03-20T05:06:01.298 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078.front.sepia.ceph.com
2024-03-20T05:06:01.298 DEBUG:teuthology.orchestra.run.smithi078:> grep smithi078.front.sepia.ceph.com /etc/hosts
2024-03-20T05:06:01.351 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78 smithi078.front.sepia.ceph.com smithi078
2024-03-20T05:06:01.352 DEBUG:teuthology.orchestra.run.smithi078:> sudo hostname smithi078
2024-03-20T05:06:01.460 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i -e 's/smithi078.front.sepia.ceph.com/smithi078/g' /etc/hosts
2024-03-20T05:06:01.490 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i -e 's/172.21.15.78/172.21.15.78/g' /etc/hosts
2024-03-20T05:06:01.558 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i -e 's/smithi078.front.sepia.ceph.com/smithi078/g' /etc/hostname
2024-03-20T05:06:01.627 DEBUG:teuthology.orchestra.run.smithi078:> sudo hostname smithi078
2024-03-20T05:06:01.694 DEBUG:teuthology.orchestra.run.smithi078:> cat /etc/os-release
2024-03-20T05:06:01.748 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="CentOS Stream"
2024-03-20T05:06:01.748 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="9"
2024-03-20T05:06:01.748 INFO:teuthology.orchestra.run.smithi078.stdout:ID="centos"
2024-03-20T05:06:01.748 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE="rhel fedora"
2024-03-20T05:06:01.748 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="9"
2024-03-20T05:06:01.748 INFO:teuthology.orchestra.run.smithi078.stdout:PLATFORM_ID="platform:el9"
2024-03-20T05:06:01.748 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="CentOS Stream 9"
2024-03-20T05:06:01.748 INFO:teuthology.orchestra.run.smithi078.stdout:ANSI_COLOR="0;31"
2024-03-20T05:06:01.748 INFO:teuthology.orchestra.run.smithi078.stdout:LOGO="fedora-logo-icon"
2024-03-20T05:06:01.748 INFO:teuthology.orchestra.run.smithi078.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-03-20T05:06:01.748 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://centos.org/"
2024-03-20T05:06:01.749 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-03-20T05:06:01.749 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-03-20T05:06:01.749 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-03-20T05:06:01.749 INFO:teuthology.provision.fog.smithi078:Deploy complete!
2024-03-20T05:06:01.750 DEBUG:teuthology.parallel:result is None
2024-03-20T05:06:02.886 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 326, in reimage_machines
    reimaged = do_update_keys(list(reimaged.keys()))[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 290, in do_update_keys
    keys_dict = misc.ssh_keyscan(machines, _raise=_raise)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1108, in ssh_keyscan
    key = _ssh_keyscan(hostname)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1145, in _ssh_keyscan
    return sorted(keys)[0]
IndexError: list index out of range
2024-03-20T05:06:02.888 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-20T05:06:03.190 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ac966f65127a405d8f53b7bc49939c25
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 326, in reimage_machines
    reimaged = do_update_keys(list(reimaged.keys()))[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 290, in do_update_keys
    keys_dict = misc.ssh_keyscan(machines, _raise=_raise)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1108, in ssh_keyscan
    key = _ssh_keyscan(hostname)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1145, in _ssh_keyscan
    return sorted(keys)[0]
IndexError: list index out of range
2024-03-20T05:06:03.192 INFO:teuthology.nuke:Checking targets against current locks
2024-03-20T05:06:03.252 DEBUG:teuthology.nuke:shortname: smithi078
2024-03-20T05:06:03.252 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-03-20T05:06:03.278 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-18_21:32:02-powercycle-squid-distro-default-smithi/7609557', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-20 05:00:54.288121', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI1iKQbAtYD730zZK7R0bQN7kk4ABYjsq4r7prSZXeGFu6ZZd+Rp7ewRJIX1wBL67laHWBuwdWNvi8rZGDL3hX0='}
2024-03-20T05:06:03.283 DEBUG:teuthology.nuke:shortname: smithi100
2024-03-20T05:06:03.283 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-03-20T05:06:03.311 DEBUG:teuthology.nuke:shortname: smithi102
2024-03-20T05:06:03.312 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-03-20T05:06:03.313 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi100.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-18_21:32:02-powercycle-squid-distro-default-smithi/7609558', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-20 05:05:55.349394', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBpKrYcdfc1iFhY2BBv1Y4fdI095ypNBlVis0eRUpAYt/5nbQCMckSwxe2+YRAtBAEI/I1IP2+eXOpD2UzIb0hc='}
2024-03-20T05:06:03.341 DEBUG:teuthology.nuke:shortname: smithi191
2024-03-20T05:06:03.341 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-03-20T05:06:03.342 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-18_21:32:02-powercycle-squid-distro-default-smithi/7609558', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-20 05:05:55.352328', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNyB11Sam/M6zeqogM2R4ACjVc0KfLhRBFc42KUm/VJ0Ic6rEMhjnwjlyJ08IX6HUc8T+mxbYq/LJ1COxAnMoMI='}
2024-03-20T05:06:03.348 INFO:teuthology.orchestra.console.smithi078:Power off
2024-03-20T05:06:03.348 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-20T05:06:03.373 DEBUG:teuthology.orchestra.console.smithi078:power off output: Chassis Power Control: Down/Off
2024-03-20T05:06:03.373 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:06:03.390 INFO:teuthology.orchestra.console.smithi100:Power off
2024-03-20T05:06:03.390 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-20T05:06:03.402 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-18_21:32:02-powercycle-squid-distro-default-smithi/7609558', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-20 05:05:55.351215', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGHC5EQUa5qfJsCvKRSanMIrdg66F+G3o7nLZfLQpQ8Lgw5gD/0TlJ+MlBZiyf0XL1b4OoI9+r4RSu5Ct1T1LvA='}
2024-03-20T05:06:03.407 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on
2024-03-20T05:06:03.413 DEBUG:teuthology.orchestra.console.smithi100:power off output: Chassis Power Control: Down/Off
2024-03-20T05:06:03.413 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:06:03.435 INFO:teuthology.orchestra.console.smithi102:Power off
2024-03-20T05:06:03.435 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-20T05:06:03.444 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is on
2024-03-20T05:06:03.457 DEBUG:teuthology.orchestra.console.smithi102:power off output: Chassis Power Control: Down/Off
2024-03-20T05:06:03.457 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:06:03.471 INFO:teuthology.orchestra.console.smithi191:Power off
2024-03-20T05:06:03.472 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-20T05:06:03.481 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on
2024-03-20T05:06:03.493 DEBUG:teuthology.orchestra.console.smithi191:power off output: Chassis Power Control: Down/Off
2024-03-20T05:06:03.493 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:06:03.514 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on
2024-03-20T05:06:07.408 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:06:07.445 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:06:07.482 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:06:07.516 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:06:07.525 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on
2024-03-20T05:06:07.555 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is on
2024-03-20T05:06:07.592 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on
2024-03-20T05:06:07.626 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on
2024-03-20T05:06:11.526 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:06:11.556 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:06:11.594 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:06:11.627 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:06:11.636 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on
2024-03-20T05:06:11.664 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is off
2024-03-20T05:06:11.703 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on
2024-03-20T05:06:11.736 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is off
2024-03-20T05:06:11.765 INFO:teuthology.orchestra.console.smithi100:Power off completed
2024-03-20T05:06:11.837 INFO:teuthology.orchestra.console.smithi191:Power off completed
2024-03-20T05:06:11.921 INFO:teuthology.lock.ops:unlocked: smithi100.front.sepia.ceph.com
2024-03-20T05:06:11.988 INFO:teuthology.lock.ops:unlocked: smithi191.front.sepia.ceph.com
2024-03-20T05:06:15.637 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:06:15.705 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-20T05:06:15.747 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is off
2024-03-20T05:06:15.816 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is off
2024-03-20T05:06:15.848 INFO:teuthology.orchestra.console.smithi078:Power off completed
2024-03-20T05:06:15.917 INFO:teuthology.orchestra.console.smithi102:Power off completed
2024-03-20T05:06:15.998 INFO:teuthology.lock.ops:unlocked: smithi078.front.sepia.ceph.com
2024-03-20T05:06:16.062 INFO:teuthology.lock.ops:unlocked: smithi102.front.sepia.ceph.com
2024-03-20T05:06:16.624 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 326, in reimage_machines
    reimaged = do_update_keys(list(reimaged.keys()))[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 290, in do_update_keys
    keys_dict = misc.ssh_keyscan(machines, _raise=_raise)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1108, in ssh_keyscan
    key = _ssh_keyscan(hostname)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1145, in _ssh_keyscan
    return sorted(keys)[0]
IndexError: list index out of range