2023-11-04T04:58:12.643 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2
2023-11-04T04:58:12.644 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T04:58:12.796 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7fe7d6b40b50>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7fe7d68eaca0>, signals=[15])
2023-11-04T04:58:12.797 INFO:teuthology.lock.ops:Start node 'smithi098.front.sepia.ceph.com' reimaging
2023-11-04T04:58:12.797 INFO:teuthology.lock.ops:Updating [smithi098.front.sepia.ceph.com]: reset os type and version on server
2023-11-04T04:58:12.798 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server
2023-11-04T04:58:12.824 INFO:teuthology.lock.ops:Start node 'smithi111.front.sepia.ceph.com' reimaging
2023-11-04T04:58:12.825 INFO:teuthology.lock.ops:Updating [smithi111.front.sepia.ceph.com]: reset os type and version on server
2023-11-04T04:58:12.826 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server
2023-11-04T04:58:12.827 DEBUG:teuthology.provision.fog.smithi098:FOG request: GET /host {"name": "smithi098"}
2023-11-04T04:58:12.848 DEBUG:teuthology.provision.fog.smithi111:FOG request: GET /host {"name": "smithi111"}
2023-11-04T04:58:12.904 DEBUG:teuthology.provision.fog.smithi098:FOG request: GET /image {"name": "smithi_ubuntu_22.04"}
2023-11-04T04:58:12.920 DEBUG:teuthology.provision.fog.smithi111:FOG request: GET /image {"name": "smithi_ubuntu_22.04"}
2023-11-04T04:58:12.945 DEBUG:teuthology.provision.fog.smithi098:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-11-04T04:58:12.946 DEBUG:teuthology.provision.fog.smithi098:FOG request: PUT /host/203 {"imageID": 6}
2023-11-04T04:58:12.963 DEBUG:teuthology.provision.fog.smithi111:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-11-04T04:58:12.963 DEBUG:teuthology.provision.fog.smithi111:FOG request: PUT /host/216 {"imageID": 6}
2023-11-04T04:58:12.994 INFO:teuthology.provision.fog.smithi098:Scheduling deploy of ubuntu 22.04
2023-11-04T04:58:12.994 DEBUG:teuthology.provision.fog.smithi098:FOG request: GET /task/active None
2023-11-04T04:58:13.012 INFO:teuthology.provision.fog.smithi111:Scheduling deploy of ubuntu 22.04
2023-11-04T04:58:13.012 DEBUG:teuthology.provision.fog.smithi111:FOG request: GET /task/active None
2023-11-04T04:58:13.074 DEBUG:teuthology.provision.fog.smithi098:FOG request: GET /tasktype {"name": "deploy"}
2023-11-04T04:58:13.078 DEBUG:teuthology.provision.fog.smithi111:FOG request: GET /tasktype {"name": "deploy"}
2023-11-04T04:58:13.116 DEBUG:teuthology.provision.fog.smithi098:FOG request: POST /host/203/task {"taskTypeID": 1}
2023-11-04T04:58:13.121 DEBUG:teuthology.provision.fog.smithi111:FOG request: POST /host/216/task {"taskTypeID": 1}
2023-11-04T04:58:13.171 DEBUG:teuthology.provision.fog.smithi111:FOG request: GET /task/active None
2023-11-04T04:58:13.176 DEBUG:teuthology.provision.fog.smithi098:FOG request: GET /task/active None
2023-11-04T04:58:13.273 INFO:teuthology.orchestra.console.smithi098:Power off
2023-11-04T04:58:13.274 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:58:13.287 INFO:teuthology.orchestra.console.smithi111:Power off
2023-11-04T04:58:13.287 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:58:33.341 DEBUG:teuthology.orchestra.console.smithi098:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:58:33.342 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:58:33.357 DEBUG:teuthology.orchestra.console.smithi111:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:58:33.358 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:58:53.413 DEBUG:teuthology.orchestra.console.smithi098:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:58:53.414 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:58:53.424 DEBUG:teuthology.orchestra.console.smithi111:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:58:53.424 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:59:13.485 DEBUG:teuthology.orchestra.console.smithi098:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:59:13.486 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:59:13.495 DEBUG:teuthology.orchestra.console.smithi111:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:59:13.496 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:59:14.494 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T04:59:14.503 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T04:59:18.498 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:59:18.505 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:59:19.608 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T04:59:19.615 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T04:59:23.610 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:59:23.617 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:59:24.720 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T04:59:24.725 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T04:59:28.725 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:59:28.732 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:59:29.834 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T04:59:29.845 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T04:59:33.838 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:59:33.846 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:59:34.948 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T04:59:34.956 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T04:59:38.953 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:59:38.960 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:59:40.062 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T04:59:40.071 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T04:59:44.066 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:59:44.073 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:59:45.176 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T04:59:45.182 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T04:59:49.180 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:59:49.188 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:59:50.290 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T04:59:50.297 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T04:59:54.294 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:59:54.302 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:59:55.404 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T04:59:55.412 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T04:59:59.409 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:59:59.417 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:00:00.520 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:00:00.526 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:00:04.522 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:00:04.529 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:00:05.631 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:00:05.638 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:00:09.635 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:00:09.644 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:00:10.747 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:00:10.754 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:00:14.750 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:00:14.757 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:00:15.860 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:00:15.865 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:00:19.862 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:00:19.870 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:00:20.974 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:00:20.980 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:00:24.978 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:00:24.985 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:00:26.088 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:00:26.089 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T05:00:26.089 ERROR:teuthology.orchestra.console.smithi098:Failed to power off
2023-11-04T05:00:26.090 INFO:teuthology.orchestra.console.smithi098:Power on
2023-11-04T05:00:26.090 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T05:00:26.098 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:00:26.098 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T05:00:26.099 ERROR:teuthology.orchestra.console.smithi111:Failed to power off
2023-11-04T05:00:26.099 INFO:teuthology.orchestra.console.smithi111:Power on
2023-11-04T05:00:26.099 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T05:00:46.162 DEBUG:teuthology.orchestra.console.smithi098:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:00:46.163 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T05:00:46.173 DEBUG:teuthology.orchestra.console.smithi111:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:00:46.174 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T05:01:06.234 DEBUG:teuthology.orchestra.console.smithi098:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:01:06.234 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T05:01:06.249 DEBUG:teuthology.orchestra.console.smithi111:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:01:06.250 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T05:01:26.304 DEBUG:teuthology.orchestra.console.smithi098:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:01:26.305 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:01:26.317 DEBUG:teuthology.orchestra.console.smithi111:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:01:26.318 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:01:27.314 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:01:27.325 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:01:31.318 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:01:31.326 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:01:32.428 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:01:32.434 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:01:36.430 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:01:36.437 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:01:37.539 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:01:37.544 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:01:41.545 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:01:41.553 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:01:42.656 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:01:42.661 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:01:46.661 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:01:46.668 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:01:47.770 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:01:47.776 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:01:51.774 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:01:51.782 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:01:52.885 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:01:52.890 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:01:56.890 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:01:56.897 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:01:57.999 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:01:58.004 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:02:02.004 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:02:02.011 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:02:03.114 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:02:03.120 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:02:07.118 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:02:07.125 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:02:08.227 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:02:08.234 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:02:12.230 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:02:12.237 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:02:13.340 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:02:13.346 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:02:17.342 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:02:17.350 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:02:18.452 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:02:18.458 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:02:22.455 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:02:22.464 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:02:23.566 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:02:23.573 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:02:27.570 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:02:27.578 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:02:28.680 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:02:28.686 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:02:32.682 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:02:32.689 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:02:33.792 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:02:33.797 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:02:37.797 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:02:37.804 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:02:38.907 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:02:38.907 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T05:02:38.908 DEBUG:teuthology.provision.fog.smithi098:Canceling deploy task with ID 582649
2023-11-04T05:02:38.908 DEBUG:teuthology.provision.fog.smithi098:FOG request: DELETE /task/cancel {"id": 582649}
2023-11-04T05:02:38.914 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:02:38.914 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T05:02:38.914 DEBUG:teuthology.provision.fog.smithi111:Canceling deploy task with ID 582648
2023-11-04T05:02:38.915 DEBUG:teuthology.provision.fog.smithi111:FOG request: DELETE /task/cancel {"id": 582648}
2023-11-04T05:02:38.967 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi098
2023-11-04T05:02:38.967 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T05:02:39.107 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=71ab3cfd06ea4173b01ecf7730115607
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi098
2023-11-04T05:02:39.109 INFO:teuthology.nuke:Checking targets against current locks
2023-11-04T05:02:39.183 DEBUG:teuthology.nuke:shortname: smithi098
2023-11-04T05:02:39.184 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-04T05:02:39.202 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-11-04_04:30:51-fs-rishabh-2023nov3-testing-default-smithi/7447106', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-04 04:58:11.827027', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP1X4qp0uCbjlqsXN3hys7R6PXJnZT6o32FKUbvB3meiIdjXz0S1Llof+BeP34T9LSUOt0joKXc9BGL6vSJNVbA='}
2023-11-04T05:02:39.211 DEBUG:teuthology.nuke:shortname: smithi111
2023-11-04T05:02:39.211 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-04T05:02:39.232 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-11-04_04:30:51-fs-rishabh-2023nov3-testing-default-smithi/7447106', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-04 04:58:11.825061', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}
2023-11-04T05:02:39.246 INFO:teuthology.orchestra.console.smithi098:Power off
2023-11-04T05:02:39.246 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T05:02:39.286 INFO:teuthology.orchestra.console.smithi111:Power off
2023-11-04T05:02:39.286 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T05:02:59.317 DEBUG:teuthology.orchestra.console.smithi098:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:02:59.318 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T05:02:59.356 DEBUG:teuthology.orchestra.console.smithi111:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:02:59.357 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T05:03:19.388 DEBUG:teuthology.orchestra.console.smithi098:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:03:19.389 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T05:03:19.424 DEBUG:teuthology.orchestra.console.smithi111:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:03:19.425 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T05:03:39.460 DEBUG:teuthology.orchestra.console.smithi098:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:03:39.461 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:03:39.496 DEBUG:teuthology.orchestra.console.smithi111:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:03:39.498 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:03:40.469 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:03:40.506 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:03:44.471 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:03:44.507 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:03:45.588 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:03:45.616 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:03:49.592 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:03:49.617 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:03:50.703 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:03:50.727 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:03:54.706 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:03:54.728 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:03:55.818 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:03:55.838 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:03:59.822 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:03:59.840 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:04:00.932 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:04:00.948 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:04:04.938 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:04:04.949 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:04:06.050 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:04:06.058 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:04:10.055 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:04:10.063 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:04:11.165 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:04:11.172 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:04:15.170 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:04:15.179 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:04:16.282 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:04:16.289 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:04:20.286 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:04:20.295 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:04:21.397 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:04:21.405 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:04:25.402 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:04:25.410 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:04:26.512 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:04:26.519 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:04:30.518 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:04:30.526 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:04:31.628 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:04:31.637 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:04:35.632 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:04:35.640 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:04:36.742 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:04:36.749 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:04:40.746 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:04:40.755 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:04:41.858 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:04:41.867 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:04:45.862 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:04:45.870 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:04:46.973 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:04:46.980 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:04:50.978 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:04:50.989 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:04:52.090 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-11-04T05:04:52.091 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T05:04:52.091 ERROR:teuthology.orchestra.console.smithi098:Failed to power off
2023-11-04T05:04:52.108 DEBUG:teuthology.orchestra.console.smithi111:check power output: 
2023-11-04T05:04:52.108 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T05:04:52.109 ERROR:teuthology.orchestra.console.smithi111:Failed to power off
2023-11-04T05:04:52.253 INFO:teuthology.lock.ops:unlocked: smithi098.front.sepia.ceph.com
2023-11-04T05:04:52.261 INFO:teuthology.lock.ops:unlocked: smithi111.front.sepia.ceph.com
2023-11-04T05:04:52.493 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi098