2023-09-21T10:44:18.772 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-21T10:44:18.774 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-09-21T10:44:18.896 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f05d26f2c70>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f05d239eca0>, signals=[15])
2023-09-21T10:44:18.898 INFO:teuthology.lock.ops:Start node 'smithi100.front.sepia.ceph.com' reimaging
2023-09-21T10:44:18.898 INFO:teuthology.lock.ops:Updating [smithi100.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T10:44:18.899 INFO:teuthology.lock.ops:Updating smithi100.front.sepia.ceph.com on lock server
2023-09-21T10:44:18.923 INFO:teuthology.lock.ops:Start node 'smithi121.front.sepia.ceph.com' reimaging
2023-09-21T10:44:18.923 INFO:teuthology.lock.ops:Updating [smithi121.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T10:44:18.923 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server
2023-09-21T10:44:18.925 DEBUG:teuthology.provision.fog.smithi100:FOG request: GET /host {"name": "smithi100"}
2023-09-21T10:44:18.947 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /host {"name": "smithi121"}
2023-09-21T10:44:19.005 DEBUG:teuthology.provision.fog.smithi100:FOG request: GET /image {"name": "smithi_ubuntu_22.04"}
2023-09-21T10:44:19.014 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /image {"name": "smithi_ubuntu_22.04"}
2023-09-21T10:44:19.049 DEBUG:teuthology.provision.fog.smithi100:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-09-21T10:44:19.049 DEBUG:teuthology.provision.fog.smithi100:FOG request: PUT /host/205 {"imageID": 6}
2023-09-21T10:44:19.060 DEBUG:teuthology.provision.fog.smithi121:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-09-21T10:44:19.060 DEBUG:teuthology.provision.fog.smithi121:FOG request: PUT /host/226 {"imageID": 6}
2023-09-21T10:44:19.098 INFO:teuthology.provision.fog.smithi100:Scheduling deploy of ubuntu 22.04
2023-09-21T10:44:19.098 DEBUG:teuthology.provision.fog.smithi100:FOG request: GET /task/active None
2023-09-21T10:44:19.101 INFO:teuthology.provision.fog.smithi121:Scheduling deploy of ubuntu 22.04
2023-09-21T10:44:19.102 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None
2023-09-21T10:44:19.170 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T10:44:19.174 DEBUG:teuthology.provision.fog.smithi100:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T10:44:19.210 DEBUG:teuthology.provision.fog.smithi100:FOG request: POST /host/205/task {"taskTypeID": 1}
2023-09-21T10:44:19.214 DEBUG:teuthology.provision.fog.smithi121:FOG request: POST /host/226/task {"taskTypeID": 1}
2023-09-21T10:44:19.272 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None
2023-09-21T10:44:19.282 DEBUG:teuthology.provision.fog.smithi100:FOG request: GET /task/active None
2023-09-21T10:44:19.391 INFO:teuthology.orchestra.console.smithi121:Power off
2023-09-21T10:44:19.392 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T10:44:19.406 INFO:teuthology.orchestra.console.smithi100:Power off
2023-09-21T10:44:19.406 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T10:44:39.464 DEBUG:teuthology.orchestra.console.smithi121:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T10:44:39.465 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T10:44:39.475 DEBUG:teuthology.orchestra.console.smithi100:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T10:44:39.476 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T10:44:59.538 DEBUG:teuthology.orchestra.console.smithi121:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T10:44:59.538 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T10:44:59.548 DEBUG:teuthology.orchestra.console.smithi100:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T10:44:59.549 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T10:45:19.605 DEBUG:teuthology.orchestra.console.smithi121:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T10:45:19.605 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:45:19.615 DEBUG:teuthology.orchestra.console.smithi100:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T10:45:19.616 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:45:20.614 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:45:20.623 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:45:24.619 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:45:24.625 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:45:25.728 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:45:25.733 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:45:29.734 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:45:29.740 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:45:30.844 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:45:30.848 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:45:34.849 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:45:34.857 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:45:35.959 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:45:35.964 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:45:39.964 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:45:39.972 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:45:41.074 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:45:41.080 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:45:45.080 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:45:45.086 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:45:46.189 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:45:46.194 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:45:50.194 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:45:50.200 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:45:51.303 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:45:51.307 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:45:55.306 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:45:55.313 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:45:56.417 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:45:56.421 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:46:00.422 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:46:00.429 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:46:01.532 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:46:01.537 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:46:05.534 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:46:05.540 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:46:06.643 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:46:06.647 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:46:10.648 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:46:10.654 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:46:11.757 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:46:11.762 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:46:15.763 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:46:15.769 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:46:16.874 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:46:16.878 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:46:20.879 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:46:20.886 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:46:21.989 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:46:21.993 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:46:25.994 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:46:26.001 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:46:27.104 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:46:27.109 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:46:31.110 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:46:31.117 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:46:32.220 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:46:32.221 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T10:46:32.221 ERROR:teuthology.orchestra.console.smithi121:Failed to power off
2023-09-21T10:46:32.221 INFO:teuthology.orchestra.console.smithi121:Power on
2023-09-21T10:46:32.222 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T10:46:32.230 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:46:32.231 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T10:46:32.231 ERROR:teuthology.orchestra.console.smithi100:Failed to power off
2023-09-21T10:46:32.231 INFO:teuthology.orchestra.console.smithi100:Power on
2023-09-21T10:46:32.231 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T10:46:52.292 DEBUG:teuthology.orchestra.console.smithi121:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T10:46:52.292 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T10:46:52.303 DEBUG:teuthology.orchestra.console.smithi100:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T10:46:52.304 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T10:47:12.360 DEBUG:teuthology.orchestra.console.smithi121:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T10:47:12.360 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T10:47:12.377 DEBUG:teuthology.orchestra.console.smithi100:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T10:47:12.377 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T10:47:32.429 DEBUG:teuthology.orchestra.console.smithi121:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T10:47:32.430 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:47:32.447 DEBUG:teuthology.orchestra.console.smithi100:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T10:47:32.448 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:47:33.439 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:47:33.454 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:47:37.442 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:47:37.455 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:47:38.552 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:47:38.562 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:47:42.557 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:47:42.564 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:47:43.666 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:47:43.672 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:47:47.670 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:47:47.676 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:47:48.778 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:47:48.784 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:47:52.782 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:47:52.788 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:47:53.890 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:47:53.896 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:47:57.894 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:47:57.900 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:47:59.004 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:47:59.008 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:48:03.006 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:48:03.013 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:48:04.116 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:48:04.121 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:48:08.118 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:48:08.124 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:48:09.226 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:48:09.232 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:48:13.230 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:48:13.237 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:48:14.341 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:48:14.347 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:48:18.346 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:48:18.352 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:48:19.454 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:48:19.459 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:48:23.458 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:48:23.464 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:48:24.567 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:48:24.573 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:48:28.570 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:48:28.576 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:48:29.678 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:48:29.684 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:48:33.682 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:48:33.689 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:48:34.791 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:48:34.797 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:48:38.794 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:48:38.801 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:48:39.903 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:48:39.908 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:48:43.907 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:48:43.915 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:48:45.018 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:48:45.018 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T10:48:45.019 DEBUG:teuthology.provision.fog.smithi121:Canceling deploy task with ID 499171
2023-09-21T10:48:45.019 DEBUG:teuthology.provision.fog.smithi121:FOG request: DELETE /task/cancel {"id": 499171}
2023-09-21T10:48:45.024 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:48:45.024 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T10:48:45.024 DEBUG:teuthology.provision.fog.smithi100:Canceling deploy task with ID 499172
2023-09-21T10:48:45.025 DEBUG:teuthology.provision.fog.smithi100:FOG request: DELETE /task/cancel {"id": 499172}
2023-09-21T10:48:45.120 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi121
2023-09-21T10:48:45.121 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-09-21T10:48:45.262 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=4e41d7525fb84255a6138f617d23e31d
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi121
2023-09-21T10:48:45.265 INFO:teuthology.nuke:Checking targets against current locks
2023-09-21T10:48:45.324 DEBUG:teuthology.nuke:shortname: smithi100
2023-09-21T10:48:45.325 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T10:48:45.347 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi100.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mkogan-2023-09-20_11:11:11-rgw-rgw-wip-t62141-valg-supp_i007-distro-default-smithi/7399442', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 10:44:17.980901', 'locked_by': 'scheduled_mkogan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDWK+DDsnn36F3dyE7RoAnqrhNL1tuuGKQNOB7wB5pQmZCY+ochzcrVfhhHITZeFdeJ0if7J71l46+Pfb1V4URk='}
2023-09-21T10:48:45.351 DEBUG:teuthology.nuke:shortname: smithi121
2023-09-21T10:48:45.351 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T10:48:45.375 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mkogan-2023-09-20_11:11:11-rgw-rgw-wip-t62141-valg-supp_i007-distro-default-smithi/7399442', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 10:44:17.979171', 'locked_by': 'scheduled_mkogan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1jn1oLBF86q1pnE7RuFXENl5s/X1tespMys9PDEb7ck5yZ2AevLKGrQjEtmL5Zl6NRsF5t8khgq2lSYP9XV8s='}
2023-09-21T10:48:45.405 INFO:teuthology.orchestra.console.smithi100:Power off
2023-09-21T10:48:45.406 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T10:48:45.444 INFO:teuthology.orchestra.console.smithi121:Power off
2023-09-21T10:48:45.445 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T10:49:05.480 DEBUG:teuthology.orchestra.console.smithi100:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T10:49:05.481 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T10:49:05.512 DEBUG:teuthology.orchestra.console.smithi121:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T10:49:05.512 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T10:49:25.548 DEBUG:teuthology.orchestra.console.smithi100:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T10:49:25.548 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T10:49:25.584 DEBUG:teuthology.orchestra.console.smithi121:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T10:49:25.584 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T10:49:45.621 DEBUG:teuthology.orchestra.console.smithi100:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T10:49:45.622 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:49:45.651 DEBUG:teuthology.orchestra.console.smithi121:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T10:49:45.652 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:49:46.631 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:49:46.660 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:49:50.634 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:49:50.661 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:49:51.743 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:49:51.769 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:49:55.749 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:49:55.770 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:49:56.858 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:49:56.879 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:50:00.862 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:50:00.880 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:50:01.970 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:50:01.988 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:50:05.976 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:50:05.989 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:50:07.086 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:50:07.096 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:50:11.090 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:50:11.097 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:50:12.199 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:50:12.204 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:50:16.202 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:50:16.209 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:50:17.310 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:50:17.317 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:50:21.315 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:50:21.323 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:50:22.426 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:50:22.432 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:50:26.431 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:50:26.439 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:50:27.542 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:50:27.547 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:50:31.548 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:50:31.554 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:50:32.657 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:50:32.662 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:50:36.662 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:50:36.670 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:50:37.772 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:50:37.777 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:50:41.778 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:50:41.785 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:50:42.889 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:50:42.893 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:50:46.892 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:50:46.899 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:50:48.001 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:50:48.008 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:50:52.006 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:50:52.013 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:50:53.116 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:50:53.122 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:50:57.122 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:50:57.129 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:50:58.231 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T10:50:58.232 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T10:50:58.233 ERROR:teuthology.orchestra.console.smithi100:Failed to power off
2023-09-21T10:50:58.237 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T10:50:58.237 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T10:50:58.238 ERROR:teuthology.orchestra.console.smithi121:Failed to power off
2023-09-21T10:50:58.383 INFO:teuthology.lock.ops:unlocked: smithi100.front.sepia.ceph.com
2023-09-21T10:50:58.390 INFO:teuthology.lock.ops:unlocked: smithi121.front.sepia.ceph.com
2023-09-21T10:50:58.614 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi121