2023-11-04T05:16:55.156 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2
2023-11-04T05:16:55.158 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T05:16:55.296 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f314697cbb0>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f3146726c10>, signals=[15])
2023-11-04T05:16:55.297 INFO:teuthology.lock.ops:Start node 'smithi102.front.sepia.ceph.com' reimaging
2023-11-04T05:16:55.298 INFO:teuthology.lock.ops:Updating [smithi102.front.sepia.ceph.com]: reset os type and version on server
2023-11-04T05:16:55.298 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server
2023-11-04T05:16:55.317 INFO:teuthology.lock.ops:Start node 'smithi116.front.sepia.ceph.com' reimaging
2023-11-04T05:16:55.317 INFO:teuthology.lock.ops:Updating [smithi116.front.sepia.ceph.com]: reset os type and version on server
2023-11-04T05:16:55.318 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server
2023-11-04T05:16:55.319 DEBUG:teuthology.provision.fog.smithi102:FOG request: GET /host {"name": "smithi102"}
2023-11-04T05:16:55.342 DEBUG:teuthology.provision.fog.smithi116:FOG request: GET /host {"name": "smithi116"}
2023-11-04T05:16:55.396 DEBUG:teuthology.provision.fog.smithi102:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-11-04T05:16:55.418 DEBUG:teuthology.provision.fog.smithi116:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-11-04T05:16:55.439 DEBUG:teuthology.provision.fog.smithi102:Requesting image smithi_rhel_8.6 (ID 5)
2023-11-04T05:16:55.440 DEBUG:teuthology.provision.fog.smithi102:FOG request: PUT /host/207 {"imageID": 5}
2023-11-04T05:16:55.463 DEBUG:teuthology.provision.fog.smithi116:Requesting image smithi_rhel_8.6 (ID 5)
2023-11-04T05:16:55.464 DEBUG:teuthology.provision.fog.smithi116:FOG request: PUT /host/221 {"imageID": 5}
2023-11-04T05:16:55.487 INFO:teuthology.provision.fog.smithi102:Scheduling deploy of rhel 8.6
2023-11-04T05:16:55.487 DEBUG:teuthology.provision.fog.smithi102:FOG request: GET /task/active None
2023-11-04T05:16:55.509 INFO:teuthology.provision.fog.smithi116:Scheduling deploy of rhel 8.6
2023-11-04T05:16:55.510 DEBUG:teuthology.provision.fog.smithi116:FOG request: GET /task/active None
2023-11-04T05:16:55.546 DEBUG:teuthology.provision.fog.smithi102:FOG request: GET /tasktype {"name": "deploy"}
2023-11-04T05:16:55.575 DEBUG:teuthology.provision.fog.smithi116:FOG request: GET /tasktype {"name": "deploy"}
2023-11-04T05:16:55.590 DEBUG:teuthology.provision.fog.smithi102:FOG request: POST /host/207/task {"taskTypeID": 1}
2023-11-04T05:16:55.615 DEBUG:teuthology.provision.fog.smithi116:FOG request: POST /host/221/task {"taskTypeID": 1}
2023-11-04T05:16:55.643 DEBUG:teuthology.provision.fog.smithi102:FOG request: GET /task/active None
2023-11-04T05:16:55.670 DEBUG:teuthology.provision.fog.smithi116:FOG request: GET /task/active None
2023-11-04T05:16:55.761 INFO:teuthology.orchestra.console.smithi116:Power off
2023-11-04T05:16:55.762 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T05:16:55.781 INFO:teuthology.orchestra.console.smithi102:Power off
2023-11-04T05:16:55.782 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T05:17:15.833 DEBUG:teuthology.orchestra.console.smithi116:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:17:15.834 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T05:17:15.849 DEBUG:teuthology.orchestra.console.smithi102:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:17:15.850 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T05:17:35.900 DEBUG:teuthology.orchestra.console.smithi116:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:17:35.900 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T05:17:35.916 DEBUG:teuthology.orchestra.console.smithi102:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:17:35.916 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T05:17:55.968 DEBUG:teuthology.orchestra.console.smithi116:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:17:55.969 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:17:55.984 DEBUG:teuthology.orchestra.console.smithi102:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:17:55.984 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:17:56.978 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:17:56.991 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:18:00.981 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:18:00.992 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:18:02.092 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:18:02.100 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:18:06.096 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:18:06.103 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:18:07.205 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:18:07.210 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:18:11.210 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:18:11.217 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:18:12.320 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:18:12.326 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:18:16.322 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:18:16.330 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:18:17.432 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:18:17.438 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:18:21.434 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:18:21.441 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:18:22.542 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:18:22.551 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:18:26.548 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:18:26.555 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:18:27.657 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:18:27.663 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:18:31.662 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:18:31.669 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:18:32.772 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:18:32.776 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:18:36.774 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:18:36.782 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:18:37.884 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:18:37.889 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:18:41.886 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:18:41.896 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:18:42.998 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:18:43.003 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:18:47.005 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:18:47.013 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:18:48.114 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:18:48.121 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:18:52.118 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:18:52.126 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:18:53.229 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:18:53.235 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:18:57.234 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:18:57.241 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:18:58.344 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:18:58.349 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:19:02.346 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:19:02.354 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:19:03.457 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:19:03.461 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:19:07.458 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:19:07.465 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:19:08.568 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:19:08.569 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T05:19:08.570 ERROR:teuthology.orchestra.console.smithi116:Failed to power off
2023-11-04T05:19:08.570 INFO:teuthology.orchestra.console.smithi116:Power on
2023-11-04T05:19:08.570 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T05:19:08.579 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:19:08.580 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T05:19:08.580 ERROR:teuthology.orchestra.console.smithi102:Failed to power off
2023-11-04T05:19:08.581 INFO:teuthology.orchestra.console.smithi102:Power on
2023-11-04T05:19:08.581 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T05:19:28.640 DEBUG:teuthology.orchestra.console.smithi116:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:19:28.641 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T05:19:28.652 DEBUG:teuthology.orchestra.console.smithi102:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:19:28.653 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T05:19:48.713 DEBUG:teuthology.orchestra.console.smithi116:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:19:48.714 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T05:19:48.722 DEBUG:teuthology.orchestra.console.smithi102:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:19:48.723 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T05:20:08.784 DEBUG:teuthology.orchestra.console.smithi116:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:20:08.785 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:20:08.793 DEBUG:teuthology.orchestra.console.smithi102:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:20:08.794 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:20:09.794 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:20:09.801 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:20:13.798 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:20:13.805 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:20:14.908 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:20:14.913 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:20:18.910 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:20:18.917 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:20:20.020 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:20:20.025 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:20:24.024 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:20:24.032 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:20:25.134 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:20:25.140 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:20:29.138 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:20:29.145 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:20:30.248 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:20:30.254 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:20:34.250 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:20:34.257 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:20:35.360 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:20:35.365 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:20:39.362 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:20:39.369 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:20:40.471 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:20:40.477 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:20:44.476 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:20:44.485 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:20:45.587 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:20:45.593 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:20:49.590 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:20:49.598 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:20:50.700 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:20:50.705 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:20:54.702 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:20:54.711 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:20:55.813 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:20:55.820 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:20:59.818 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:20:59.825 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:21:00.928 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:21:00.933 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:21:04.930 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:21:04.937 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:21:06.040 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:21:06.045 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:21:10.042 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:21:10.049 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:21:11.150 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:21:11.157 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:21:15.154 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:21:15.161 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:21:16.263 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:21:16.271 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:21:20.268 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:21:20.278 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:21:21.381 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:21:21.382 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T05:21:21.382 DEBUG:teuthology.provision.fog.smithi116:Canceling deploy task with ID 582708
2023-11-04T05:21:21.382 DEBUG:teuthology.provision.fog.smithi116:FOG request: DELETE /task/cancel {"id": 582708}
2023-11-04T05:21:21.388 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:21:21.388 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T05:21:21.389 DEBUG:teuthology.provision.fog.smithi102:Canceling deploy task with ID 582707
2023-11-04T05:21:21.389 DEBUG:teuthology.provision.fog.smithi102:FOG request: DELETE /task/cancel {"id": 582707}
2023-11-04T05:21:21.436 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi116
2023-11-04T05:21:21.436 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T05:21:21.582 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=429c3abea8e74adfb7f51f977b850f42
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi116
2023-11-04T05:21:21.584 INFO:teuthology.nuke:Checking targets against current locks
2023-11-04T05:21:21.647 DEBUG:teuthology.nuke:shortname: smithi102
2023-11-04T05:21:21.647 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-04T05:21:21.670 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-11-04_04:30:51-fs-rishabh-2023nov3-testing-default-smithi/7447131', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-04 05:16:54.322820', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDolI1f/8OEvHXCzr/jWWDOcgGaGul0yhXs02+GQeO5LPIKLiVBHdO6wa5C2uVkqIoT2yZideWvronzdoQEjyZA='}
2023-11-04T05:21:21.674 DEBUG:teuthology.nuke:shortname: smithi116
2023-11-04T05:21:21.675 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-04T05:21:21.696 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-11-04_04:30:51-fs-rishabh-2023nov3-testing-default-smithi/7447131', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-04 05:16:54.324355', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOQx0T9Cy3PuRDcKCphGZ9Vc5tqz8pcCkm3+TX+0iQMhDeU+6rMaTDJbsjF6dAqRzIoF4739qQAYE59ailaAJfs='}
2023-11-04T05:21:21.727 INFO:teuthology.orchestra.console.smithi102:Power off
2023-11-04T05:21:21.727 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T05:21:21.769 INFO:teuthology.orchestra.console.smithi116:Power off
2023-11-04T05:21:21.769 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T05:21:41.803 DEBUG:teuthology.orchestra.console.smithi102:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:21:41.805 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T05:21:41.840 DEBUG:teuthology.orchestra.console.smithi116:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:21:41.840 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T05:22:01.877 DEBUG:teuthology.orchestra.console.smithi102:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:22:01.878 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T05:22:01.913 DEBUG:teuthology.orchestra.console.smithi116:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:22:01.914 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T05:22:21.946 DEBUG:teuthology.orchestra.console.smithi102:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:22:21.947 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:22:21.989 DEBUG:teuthology.orchestra.console.smithi116:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:22:21.990 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:22:22.957 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:22:22.999 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:22:26.959 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:22:27.001 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:22:28.073 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:22:28.111 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:22:32.078 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:22:32.113 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:22:33.188 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:22:33.222 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:22:37.190 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:22:37.223 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:22:38.301 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:22:38.333 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:22:42.307 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:22:42.334 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:22:43.417 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:22:43.444 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:22:47.422 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:22:47.445 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:22:48.533 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:22:48.553 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:22:52.538 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:22:52.555 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:22:53.651 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:22:53.664 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:22:57.654 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:22:57.665 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:22:58.765 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:22:58.774 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:23:02.770 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:23:02.778 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:23:03.880 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:23:03.888 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:23:07.882 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:23:07.889 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:23:08.992 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:23:08.997 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:23:12.994 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:23:13.004 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:23:14.106 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:23:14.112 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:23:18.110 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:23:18.119 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:23:19.221 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:23:19.228 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:23:23.227 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:23:23.235 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:23:24.338 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:23:24.344 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:23:28.342 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:23:28.349 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:23:29.452 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:23:29.457 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:23:33.456 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:23:33.466 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:23:34.569 DEBUG:teuthology.orchestra.console.smithi102:check power output: 
2023-11-04T05:23:34.570 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T05:23:34.571 ERROR:teuthology.orchestra.console.smithi102:Failed to power off
2023-11-04T05:23:34.578 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:23:34.579 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T05:23:34.579 ERROR:teuthology.orchestra.console.smithi116:Failed to power off
2023-11-04T05:23:34.728 INFO:teuthology.lock.ops:unlocked: smithi102.front.sepia.ceph.com
2023-11-04T05:23:34.737 INFO:teuthology.lock.ops:unlocked: smithi116.front.sepia.ceph.com
2023-11-04T05:23:34.849 ERROR:teuthology.dispatcher.supervisor:Reimage failed 10 times ... marking machine down
2023-11-04T05:23:34.955 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi116