2023-11-04T04:39:24.075 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2
2023-11-04T04:39:24.076 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T04:39:24.255 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7fa9ddb00b50>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7fa9dd3b2d30>, signals=[15])
2023-11-04T04:39:24.258 INFO:teuthology.lock.ops:Start node 'smithi114.front.sepia.ceph.com' reimaging
2023-11-04T04:39:24.258 INFO:teuthology.lock.ops:Updating [smithi114.front.sepia.ceph.com]: reset os type and version on server
2023-11-04T04:39:24.259 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server
2023-11-04T04:39:24.283 INFO:teuthology.lock.ops:Start node 'smithi118.front.sepia.ceph.com' reimaging
2023-11-04T04:39:24.284 INFO:teuthology.lock.ops:Updating [smithi118.front.sepia.ceph.com]: reset os type and version on server
2023-11-04T04:39:24.284 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server
2023-11-04T04:39:24.285 DEBUG:teuthology.provision.fog.smithi114:FOG request: GET /host {"name": "smithi114"}
2023-11-04T04:39:24.308 INFO:teuthology.lock.ops:Start node 'smithi119.front.sepia.ceph.com' reimaging
2023-11-04T04:39:24.309 INFO:teuthology.lock.ops:Updating [smithi119.front.sepia.ceph.com]: reset os type and version on server
2023-11-04T04:39:24.309 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server
2023-11-04T04:39:24.311 DEBUG:teuthology.provision.fog.smithi118:FOG request: GET /host {"name": "smithi118"}
2023-11-04T04:39:24.333 DEBUG:teuthology.provision.fog.smithi119:FOG request: GET /host {"name": "smithi119"}
2023-11-04T04:39:24.367 DEBUG:teuthology.provision.fog.smithi114:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-11-04T04:39:24.389 DEBUG:teuthology.provision.fog.smithi118:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-11-04T04:39:24.402 DEBUG:teuthology.provision.fog.smithi119:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-11-04T04:39:24.410 DEBUG:teuthology.provision.fog.smithi114:Requesting image smithi_rhel_8.6 (ID 5)
2023-11-04T04:39:24.410 DEBUG:teuthology.provision.fog.smithi114:FOG request: PUT /host/219 {"imageID": 5}
2023-11-04T04:39:24.444 DEBUG:teuthology.provision.fog.smithi118:Requesting image smithi_rhel_8.6 (ID 5)
2023-11-04T04:39:24.444 DEBUG:teuthology.provision.fog.smithi118:FOG request: PUT /host/223 {"imageID": 5}
2023-11-04T04:39:24.449 DEBUG:teuthology.provision.fog.smithi119:Requesting image smithi_rhel_8.6 (ID 5)
2023-11-04T04:39:24.450 DEBUG:teuthology.provision.fog.smithi119:FOG request: PUT /host/224 {"imageID": 5}
2023-11-04T04:39:24.465 INFO:teuthology.provision.fog.smithi114:Scheduling deploy of rhel 8.6
2023-11-04T04:39:24.466 DEBUG:teuthology.provision.fog.smithi114:FOG request: GET /task/active None
2023-11-04T04:39:24.488 INFO:teuthology.provision.fog.smithi118:Scheduling deploy of rhel 8.6
2023-11-04T04:39:24.489 DEBUG:teuthology.provision.fog.smithi118:FOG request: GET /task/active None
2023-11-04T04:39:24.500 INFO:teuthology.provision.fog.smithi119:Scheduling deploy of rhel 8.6
2023-11-04T04:39:24.500 DEBUG:teuthology.provision.fog.smithi119:FOG request: GET /task/active None
2023-11-04T04:39:24.542 DEBUG:teuthology.provision.fog.smithi114:FOG request: GET /tasktype {"name": "deploy"}
2023-11-04T04:39:24.552 DEBUG:teuthology.provision.fog.smithi118:FOG request: GET /tasktype {"name": "deploy"}
2023-11-04T04:39:24.581 DEBUG:teuthology.provision.fog.smithi114:FOG request: POST /host/219/task {"taskTypeID": 1}
2023-11-04T04:39:24.585 DEBUG:teuthology.provision.fog.smithi119:FOG request: GET /tasktype {"name": "deploy"}
2023-11-04T04:39:24.602 DEBUG:teuthology.provision.fog.smithi118:FOG request: POST /host/223/task {"taskTypeID": 1}
2023-11-04T04:39:24.625 DEBUG:teuthology.provision.fog.smithi119:FOG request: POST /host/224/task {"taskTypeID": 1}
2023-11-04T04:39:24.644 DEBUG:teuthology.provision.fog.smithi114:FOG request: GET /task/active None
2023-11-04T04:39:24.676 DEBUG:teuthology.provision.fog.smithi118:FOG request: GET /task/active None
2023-11-04T04:39:24.696 DEBUG:teuthology.provision.fog.smithi119:FOG request: GET /task/active None
2023-11-04T04:39:24.750 INFO:teuthology.orchestra.console.smithi114:Power off
2023-11-04T04:39:24.752 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:39:24.801 INFO:teuthology.orchestra.console.smithi118:Power off
2023-11-04T04:39:24.802 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:39:24.817 INFO:teuthology.orchestra.console.smithi119:Power off
2023-11-04T04:39:24.818 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:39:44.828 DEBUG:teuthology.orchestra.console.smithi114:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:39:44.829 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:39:44.868 DEBUG:teuthology.orchestra.console.smithi118:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:39:44.869 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:39:44.888 DEBUG:teuthology.orchestra.console.smithi119:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:39:44.889 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:40:04.896 DEBUG:teuthology.orchestra.console.smithi114:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:40:04.897 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:40:04.940 DEBUG:teuthology.orchestra.console.smithi118:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:40:04.941 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:40:04.958 DEBUG:teuthology.orchestra.console.smithi119:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:40:04.959 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:40:24.965 DEBUG:teuthology.orchestra.console.smithi114:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:40:24.966 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:40:25.013 DEBUG:teuthology.orchestra.console.smithi118:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:40:25.013 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:40:25.032 DEBUG:teuthology.orchestra.console.smithi119:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:40:25.032 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:40:25.976 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:40:26.022 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:40:26.041 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:40:29.978 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:40:30.023 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:40:30.042 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:40:31.088 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:40:31.134 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:40:31.153 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:40:35.090 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:40:35.135 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:40:35.153 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:40:36.201 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:40:36.244 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:40:36.262 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:40:40.203 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:40:40.245 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:40:40.263 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:40:41.314 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:40:41.356 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:40:41.374 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:40:45.318 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:40:45.356 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:40:45.377 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:40:46.429 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:40:46.466 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:40:46.486 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:40:50.430 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:40:50.466 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:40:50.488 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:40:51.541 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:40:51.578 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:40:51.596 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:40:55.542 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:40:55.579 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:40:55.597 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:40:56.651 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:40:56.688 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:40:56.706 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:41:00.655 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:41:00.690 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:41:00.707 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:41:01.773 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:41:01.802 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:41:01.819 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:41:05.778 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:41:05.804 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:41:05.821 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:41:06.889 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:41:06.913 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:41:06.930 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:41:10.894 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:41:10.914 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:41:10.931 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:41:12.005 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:41:12.023 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:41:12.040 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:41:16.010 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:41:16.024 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:41:16.041 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:41:17.120 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:41:17.132 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:41:17.149 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:41:21.125 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:41:21.134 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:41:21.150 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:41:22.234 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:41:22.243 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:41:22.258 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:41:26.238 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:41:26.246 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:41:26.259 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:41:27.348 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:41:27.355 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:41:27.368 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:41:31.350 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:41:31.357 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:41:31.369 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:41:32.460 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:41:32.465 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:41:32.477 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:41:36.462 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:41:36.469 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:41:36.479 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:41:37.572 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:41:37.572 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T04:41:37.573 ERROR:teuthology.orchestra.console.smithi114:Failed to power off
2023-11-04T04:41:37.573 INFO:teuthology.orchestra.console.smithi114:Power on
2023-11-04T04:41:37.574 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T04:41:37.581 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:41:37.581 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T04:41:37.581 ERROR:teuthology.orchestra.console.smithi118:Failed to power off
2023-11-04T04:41:37.582 INFO:teuthology.orchestra.console.smithi118:Power on
2023-11-04T04:41:37.582 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T04:41:37.589 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:41:37.590 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T04:41:37.590 ERROR:teuthology.orchestra.console.smithi119:Failed to power off
2023-11-04T04:41:37.590 INFO:teuthology.orchestra.console.smithi119:Power on
2023-11-04T04:41:37.590 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T04:41:57.641 DEBUG:teuthology.orchestra.console.smithi114:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:41:57.642 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T04:41:57.652 DEBUG:teuthology.orchestra.console.smithi118:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:41:57.653 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T04:41:57.664 DEBUG:teuthology.orchestra.console.smithi119:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:41:57.665 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T04:42:17.713 DEBUG:teuthology.orchestra.console.smithi114:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:42:17.714 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T04:42:17.722 DEBUG:teuthology.orchestra.console.smithi118:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:42:17.723 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T04:42:17.731 DEBUG:teuthology.orchestra.console.smithi119:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:42:17.732 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T04:42:37.785 DEBUG:teuthology.orchestra.console.smithi114:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:42:37.786 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:42:37.795 DEBUG:teuthology.orchestra.console.smithi118:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:42:37.796 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:42:37.807 DEBUG:teuthology.orchestra.console.smithi119:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:42:37.808 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:42:38.794 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:42:38.806 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:42:38.817 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:42:42.799 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:42:42.809 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:42:42.818 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:42:43.912 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:42:43.917 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:42:43.926 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:42:47.916 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:42:47.923 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:42:47.930 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:42:49.026 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:42:49.032 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:42:49.039 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:42:53.031 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:42:53.040 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:42:53.048 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:42:54.142 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:42:54.150 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:42:54.159 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:42:58.146 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:42:58.153 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:42:58.160 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:42:59.257 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:42:59.261 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:42:59.270 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:43:03.262 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:03.270 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:03.278 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:04.372 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:43:04.378 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:43:04.386 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:43:08.374 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:08.382 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:08.389 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:09.485 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:43:09.490 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:43:09.498 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:43:13.490 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:13.500 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:13.508 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:14.602 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:43:14.609 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:43:14.618 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:43:18.606 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:18.616 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:18.623 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:19.718 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:43:19.724 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:43:19.731 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:43:23.723 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:23.730 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:23.737 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:24.833 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:43:24.838 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:43:24.847 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:43:28.834 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:28.842 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:28.849 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:29.944 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:43:29.950 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:43:29.957 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:43:33.946 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:33.954 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:33.961 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:35.057 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:43:35.062 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:43:35.069 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:43:39.060 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:39.067 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:39.074 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:40.170 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:43:40.175 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:43:40.184 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:43:44.174 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:44.181 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:44.193 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:45.284 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:43:45.294 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:43:45.304 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:43:49.287 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:49.295 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:49.305 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:43:50.397 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:43:50.398 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T04:43:50.398 DEBUG:teuthology.provision.fog.smithi114:Canceling deploy task with ID 582553
2023-11-04T04:43:50.399 DEBUG:teuthology.provision.fog.smithi114:FOG request: DELETE /task/cancel {"id": 582553}
2023-11-04T04:43:50.403 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:43:50.403 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T04:43:50.404 DEBUG:teuthology.provision.fog.smithi118:Canceling deploy task with ID 582554
2023-11-04T04:43:50.404 DEBUG:teuthology.provision.fog.smithi118:FOG request: DELETE /task/cancel {"id": 582554}
2023-11-04T04:43:50.416 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:43:50.417 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T04:43:50.417 DEBUG:teuthology.provision.fog.smithi119:Canceling deploy task with ID 582555
2023-11-04T04:43:50.417 DEBUG:teuthology.provision.fog.smithi119:FOG request: DELETE /task/cancel {"id": 582555}
2023-11-04T04:43:50.468 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi118
2023-11-04T04:43:50.469 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T04:43:50.621 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=a9ceb2cb620645b28c60fe1f0d295c2f
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi118
2023-11-04T04:43:50.623 INFO:teuthology.nuke:Checking targets against current locks
2023-11-04T04:43:50.690 DEBUG:teuthology.nuke:shortname: smithi114
2023-11-04T04:43:50.690 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-04T04:43:50.711 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-11-04_04:30:51-fs-rishabh-2023nov3-testing-default-smithi/7447065', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-04 04:39:23.207162', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}
2023-11-04T04:43:50.718 DEBUG:teuthology.nuke:shortname: smithi118
2023-11-04T04:43:50.718 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-04T04:43:50.740 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi118.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-11-04_04:30:51-fs-rishabh-2023nov3-testing-default-smithi/7447065', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-04 04:39:23.209905', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGSAbI2Ig6BwZRpskEhs3tpbYd/9yc8v4JQ0YRXusGSIXAkyn0DmpicGQSc9h7zmCGdPlqt78apSzfkIYpDMPio='}
2023-11-04T04:43:50.744 DEBUG:teuthology.nuke:shortname: smithi119
2023-11-04T04:43:50.744 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-04T04:43:50.767 INFO:teuthology.orchestra.console.smithi114:Power off
2023-11-04T04:43:50.768 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:43:50.777 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-11-04_04:30:51-fs-rishabh-2023nov3-testing-default-smithi/7447065', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-04 04:39:23.208899', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIv1z74QtOhtPoVF29QvLoawG3xrUUzV8r4y7zjFLrL+eGTe2lm7r1do1iNgmk+6r0juV+UJsG+lkbh3vHPz6y4='}
2023-11-04T04:43:50.825 INFO:teuthology.orchestra.console.smithi119:Power off
2023-11-04T04:43:50.826 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:43:50.846 INFO:teuthology.orchestra.console.smithi118:Power off
2023-11-04T04:43:50.847 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:44:10.836 DEBUG:teuthology.orchestra.console.smithi114:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:44:10.837 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:44:10.897 DEBUG:teuthology.orchestra.console.smithi119:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:44:10.898 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:44:10.916 DEBUG:teuthology.orchestra.console.smithi118:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:44:10.917 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:44:30.905 DEBUG:teuthology.orchestra.console.smithi114:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:44:30.906 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:44:30.968 DEBUG:teuthology.orchestra.console.smithi119:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:44:30.969 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:44:30.992 DEBUG:teuthology.orchestra.console.smithi118:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:44:30.993 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:44:50.976 DEBUG:teuthology.orchestra.console.smithi114:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:44:50.977 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:44:51.036 DEBUG:teuthology.orchestra.console.smithi119:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:44:51.037 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:44:51.060 DEBUG:teuthology.orchestra.console.smithi118:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:44:51.061 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:44:51.986 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:44:52.045 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:44:52.068 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:44:55.988 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:44:56.046 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:44:56.070 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:44:57.099 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:44:57.155 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:44:57.179 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:45:01.102 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:01.156 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:01.181 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:02.214 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:45:02.266 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:45:02.290 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:45:06.218 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:06.268 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:06.291 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:07.329 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:45:07.378 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:45:07.400 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:45:11.334 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:11.380 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:11.401 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:12.444 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:45:12.488 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:45:12.510 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:45:16.446 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:16.489 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:16.511 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:17.557 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:45:17.599 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:45:17.620 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:45:21.562 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:21.601 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:21.621 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:22.672 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:45:22.710 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:45:22.730 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:45:26.674 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:26.711 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:26.731 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:27.785 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:45:27.822 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:45:27.841 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:45:31.790 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:31.824 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:31.842 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:32.900 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:45:32.933 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:45:32.951 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:45:36.905 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:36.934 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:36.952 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:38.017 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:45:38.043 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:45:38.061 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:45:42.022 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:42.044 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:42.063 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:43.132 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:45:43.153 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:45:43.171 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:45:47.134 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:47.153 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:47.173 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:48.245 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:45:48.262 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:45:48.282 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:45:52.250 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:52.264 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:52.284 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:53.361 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:45:53.373 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:45:53.395 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:45:57.364 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:57.374 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:57.396 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:45:58.475 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:45:58.485 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:45:58.504 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:46:02.478 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:46:02.486 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:46:02.506 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:46:03.589 DEBUG:teuthology.orchestra.console.smithi114:check power output: 
2023-11-04T04:46:03.590 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T04:46:03.591 ERROR:teuthology.orchestra.console.smithi114:Failed to power off
2023-11-04T04:46:03.598 DEBUG:teuthology.orchestra.console.smithi119:check power output: 
2023-11-04T04:46:03.599 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T04:46:03.600 ERROR:teuthology.orchestra.console.smithi119:Failed to power off
2023-11-04T04:46:03.616 DEBUG:teuthology.orchestra.console.smithi118:check power output: 
2023-11-04T04:46:03.616 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T04:46:03.616 ERROR:teuthology.orchestra.console.smithi118:Failed to power off
2023-11-04T04:46:03.752 INFO:teuthology.lock.ops:unlocked: smithi114.front.sepia.ceph.com
2023-11-04T04:46:03.761 INFO:teuthology.lock.ops:unlocked: smithi119.front.sepia.ceph.com
2023-11-04T04:46:03.769 INFO:teuthology.lock.ops:unlocked: smithi118.front.sepia.ceph.com
2023-11-04T04:46:03.916 ERROR:teuthology.dispatcher.supervisor:Reimage failed 10 times ... marking machine down
2023-11-04T04:46:04.057 ERROR:teuthology.dispatcher.supervisor:Reimage failed 10 times ... marking machine down
2023-11-04T04:46:04.191 ERROR:teuthology.dispatcher.supervisor:Reimage failed 10 times ... marking machine down
2023-11-04T04:46:04.195 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi118