2023-11-04T03:47:21.278 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2
2023-11-04T03:47:21.280 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:47:21.449 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7fc72de25a60>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7fc72e92a670>, signals=[15])
2023-11-04T03:47:21.450 INFO:teuthology.lock.ops:Start node 'smithi083.front.sepia.ceph.com' reimaging
2023-11-04T03:47:21.451 INFO:teuthology.lock.ops:Updating [smithi083.front.sepia.ceph.com]: reset os type and version on server
2023-11-04T03:47:21.451 INFO:teuthology.lock.ops:Updating smithi083.front.sepia.ceph.com on lock server
2023-11-04T03:47:21.476 INFO:teuthology.lock.ops:Start node 'smithi124.front.sepia.ceph.com' reimaging
2023-11-04T03:47:21.477 INFO:teuthology.lock.ops:Updating [smithi124.front.sepia.ceph.com]: reset os type and version on server
2023-11-04T03:47:21.477 INFO:teuthology.lock.ops:Updating smithi124.front.sepia.ceph.com on lock server
2023-11-04T03:47:21.478 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /host {"name": "smithi083"}
2023-11-04T03:47:21.501 INFO:teuthology.lock.ops:Start node 'smithi136.front.sepia.ceph.com' reimaging
2023-11-04T03:47:21.501 INFO:teuthology.lock.ops:Updating [smithi136.front.sepia.ceph.com]: reset os type and version on server
2023-11-04T03:47:21.501 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server
2023-11-04T03:47:21.503 DEBUG:teuthology.provision.fog.smithi124:FOG request: GET /host {"name": "smithi124"}
2023-11-04T03:47:21.526 DEBUG:teuthology.provision.fog.smithi136:FOG request: GET /host {"name": "smithi136"}
2023-11-04T03:47:21.570 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /image {"name": "smithi_ubuntu_20.04"}
2023-11-04T03:47:21.590 DEBUG:teuthology.provision.fog.smithi124:FOG request: GET /image {"name": "smithi_ubuntu_20.04"}
2023-11-04T03:47:21.597 DEBUG:teuthology.provision.fog.smithi136:FOG request: GET /image {"name": "smithi_ubuntu_20.04"}
2023-11-04T03:47:21.621 DEBUG:teuthology.provision.fog.smithi083:Requesting image smithi_ubuntu_20.04 (ID 2)
2023-11-04T03:47:21.622 DEBUG:teuthology.provision.fog.smithi083:FOG request: PUT /host/188 {"imageID": 2}
2023-11-04T03:47:21.638 DEBUG:teuthology.provision.fog.smithi124:Requesting image smithi_ubuntu_20.04 (ID 2)
2023-11-04T03:47:21.638 DEBUG:teuthology.provision.fog.smithi124:FOG request: PUT /host/229 {"imageID": 2}
2023-11-04T03:47:21.642 DEBUG:teuthology.provision.fog.smithi136:Requesting image smithi_ubuntu_20.04 (ID 2)
2023-11-04T03:47:21.642 DEBUG:teuthology.provision.fog.smithi136:FOG request: PUT /host/241 {"imageID": 2}
2023-11-04T03:47:21.673 INFO:teuthology.provision.fog.smithi083:Scheduling deploy of ubuntu 20.04
2023-11-04T03:47:21.674 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None
2023-11-04T03:47:21.682 INFO:teuthology.provision.fog.smithi124:Scheduling deploy of ubuntu 20.04
2023-11-04T03:47:21.683 DEBUG:teuthology.provision.fog.smithi124:FOG request: GET /task/active None
2023-11-04T03:47:21.686 INFO:teuthology.provision.fog.smithi136:Scheduling deploy of ubuntu 20.04
2023-11-04T03:47:21.687 DEBUG:teuthology.provision.fog.smithi136:FOG request: GET /task/active None
2023-11-04T03:47:21.739 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /tasktype {"name": "deploy"}
2023-11-04T03:47:21.744 DEBUG:teuthology.provision.fog.smithi136:FOG request: GET /tasktype {"name": "deploy"}
2023-11-04T03:47:21.761 DEBUG:teuthology.provision.fog.smithi124:FOG request: GET /tasktype {"name": "deploy"}
2023-11-04T03:47:21.779 DEBUG:teuthology.provision.fog.smithi083:FOG request: POST /host/188/task {"taskTypeID": 1}
2023-11-04T03:47:21.784 DEBUG:teuthology.provision.fog.smithi136:FOG request: POST /host/241/task {"taskTypeID": 1}
2023-11-04T03:47:21.798 DEBUG:teuthology.provision.fog.smithi124:FOG request: POST /host/229/task {"taskTypeID": 1}
2023-11-04T03:47:21.834 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None
2023-11-04T03:47:21.839 DEBUG:teuthology.provision.fog.smithi136:FOG request: GET /task/active None
2023-11-04T03:47:21.844 DEBUG:teuthology.provision.fog.smithi124:FOG request: GET /task/active None
2023-11-04T03:47:21.949 INFO:teuthology.orchestra.console.smithi124:Power off
2023-11-04T03:47:21.950 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T03:47:21.965 INFO:teuthology.orchestra.console.smithi136:Power off
2023-11-04T03:47:21.965 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T03:47:21.978 INFO:teuthology.orchestra.console.smithi083:Power off
2023-11-04T03:47:21.979 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T03:47:21.991 DEBUG:teuthology.orchestra.console.smithi136:power off output: Chassis Power Control: Down/Off
2023-11-04T03:47:21.991 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:47:22.012 DEBUG:teuthology.orchestra.console.smithi083:power off output: Chassis Power Control: Down/Off
2023-11-04T03:47:22.013 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:47:22.020 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on
2023-11-04T03:47:22.037 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on
2023-11-04T03:47:26.023 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:47:26.040 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:47:26.134 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on
2023-11-04T03:47:26.151 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on
2023-11-04T03:47:30.138 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:47:30.152 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:47:30.248 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on
2023-11-04T03:47:30.261 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on
2023-11-04T03:47:34.250 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:47:34.262 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:47:34.361 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is off
2023-11-04T03:47:34.371 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is off
2023-11-04T03:47:34.462 INFO:teuthology.orchestra.console.smithi136:Power off completed
2023-11-04T03:47:34.472 INFO:teuthology.orchestra.console.smithi083:Power off completed
2023-11-04T03:47:34.564 INFO:teuthology.orchestra.console.smithi136:Power on
2023-11-04T03:47:34.564 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T03:47:34.573 INFO:teuthology.orchestra.console.smithi083:Power on
2023-11-04T03:47:34.574 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T03:47:34.588 DEBUG:teuthology.orchestra.console.smithi136:power on output: Chassis Power Control: Up/On
2023-11-04T03:47:34.588 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:47:34.597 DEBUG:teuthology.orchestra.console.smithi083:power on output: Chassis Power Control: Up/On
2023-11-04T03:47:34.598 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:47:34.610 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is off
2023-11-04T03:47:34.620 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is off
2023-11-04T03:47:38.614 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:47:38.621 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:47:38.724 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is off
2023-11-04T03:47:38.729 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is off
2023-11-04T03:47:42.020 DEBUG:teuthology.orchestra.console.smithi124:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T03:47:42.020 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T03:47:42.726 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:47:42.734 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:47:42.836 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on
2023-11-04T03:47:42.845 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on
2023-11-04T03:47:42.939 INFO:teuthology.orchestra.console.smithi136:Power on completed
2023-11-04T03:47:42.946 INFO:teuthology.orchestra.console.smithi083:Power on completed
2023-11-04T03:47:43.039 INFO:teuthology.provision.fog.smithi136:Waiting for deploy to finish
2023-11-04T03:47:43.040 DEBUG:teuthology.provision.fog.smithi136:FOG request: GET /task/active None
2023-11-04T03:47:43.048 INFO:teuthology.provision.fog.smithi083:Waiting for deploy to finish
2023-11-04T03:47:43.049 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None
2023-11-04T03:47:58.134 DEBUG:teuthology.provision.fog.smithi136:FOG request: GET /task/active None
2023-11-04T03:47:58.136 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None
2023-11-04T03:48:02.092 DEBUG:teuthology.orchestra.console.smithi124:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T03:48:02.092 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T03:48:13.230 DEBUG:teuthology.provision.fog.smithi136:FOG request: GET /task/active None
2023-11-04T03:48:13.233 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None
2023-11-04T03:48:22.160 DEBUG:teuthology.orchestra.console.smithi124:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T03:48:22.161 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:48:23.169 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:48:27.170 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:48:28.279 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:48:28.337 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None
2023-11-04T03:48:28.341 DEBUG:teuthology.provision.fog.smithi136:FOG request: GET /task/active None
2023-11-04T03:48:32.282 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:48:33.392 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:48:37.397 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:48:38.506 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:48:42.510 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:48:43.426 DEBUG:teuthology.provision.fog.smithi136:FOG request: GET /task/active None
2023-11-04T03:48:43.430 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None
2023-11-04T03:48:43.618 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:48:47.622 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:48:48.733 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:48:52.734 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:48:53.843 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:48:57.846 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:48:58.511 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None
2023-11-04T03:48:58.516 DEBUG:teuthology.provision.fog.smithi136:FOG request: GET /task/active None
2023-11-04T03:48:58.956 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:49:02.958 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:49:04.068 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:49:08.073 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:49:09.182 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:49:13.186 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:49:13.599 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None
2023-11-04T03:49:13.603 DEBUG:teuthology.provision.fog.smithi136:FOG request: GET /task/active None
2023-11-04T03:49:14.296 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:49:18.298 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:49:19.406 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:49:23.410 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:49:24.518 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:49:28.522 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:49:28.681 DEBUG:teuthology.provision.fog.smithi136:FOG request: GET /task/active None
2023-11-04T03:49:28.685 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None
2023-11-04T03:49:29.632 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:49:33.636 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:49:34.745 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:49:34.746 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T03:49:34.747 ERROR:teuthology.orchestra.console.smithi124:Failed to power off
2023-11-04T03:49:34.747 INFO:teuthology.orchestra.console.smithi124:Power on
2023-11-04T03:49:34.747 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T03:49:43.786 DEBUG:teuthology.provision.fog.smithi136:FOG request: GET /task/active None
2023-11-04T03:49:43.789 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None
2023-11-04T03:49:54.816 DEBUG:teuthology.orchestra.console.smithi124:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T03:49:54.816 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T03:49:58.887 DEBUG:teuthology.provision.fog.smithi136:FOG request: GET /task/active None
2023-11-04T03:49:58.889 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None
2023-11-04T03:50:13.978 DEBUG:teuthology.provision.fog.smithi136:FOG request: GET /task/active None
2023-11-04T03:50:13.980 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None
2023-11-04T03:50:14.889 DEBUG:teuthology.orchestra.console.smithi124:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T03:50:14.889 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T03:50:29.082 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None
2023-11-04T03:50:29.085 DEBUG:teuthology.provision.fog.smithi136:FOG request: GET /task/active None
2023-11-04T03:50:34.961 DEBUG:teuthology.orchestra.console.smithi124:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T03:50:34.962 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:50:35.970 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:50:39.975 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:50:41.085 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:50:44.174 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None
2023-11-04T03:50:44.176 DEBUG:teuthology.provision.fog.smithi136:FOG request: GET /task/active None
2023-11-04T03:50:45.087 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:50:46.196 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:50:50.198 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:50:51.307 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:50:55.310 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:50:56.418 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:50:59.250 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None
2023-11-04T03:50:59.254 DEBUG:teuthology.provision.fog.smithi136:FOG request: GET /task/active None
2023-11-04T03:51:00.420 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:51:01.528 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:51:05.530 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:51:06.638 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:51:10.642 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:51:11.750 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:51:14.338 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None
2023-11-04T03:51:14.340 DEBUG:teuthology.provision.fog.smithi136:FOG request: GET /task/active None
2023-11-04T03:51:15.753 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:51:16.862 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:51:20.866 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:51:21.976 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:51:25.980 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:51:27.089 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:51:29.426 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None
2023-11-04T03:51:29.430 DEBUG:teuthology.provision.fog.smithi136:FOG request: GET /task/active None
2023-11-04T03:51:29.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:51:29.509 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:51:31.093 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:51:32.203 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:51:32.570 WARNING:teuthology.provision.fog.smithi136:[Errno None] Unable to connect to port 22 on 172.21.15.136
2023-11-04T03:51:32.572 WARNING:teuthology.provision.fog.smithi083:[Errno None] Unable to connect to port 22 on 172.21.15.83
2023-11-04T03:51:36.206 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:51:37.314 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:51:38.574 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:51:38.576 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:51:41.317 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:51:41.626 WARNING:teuthology.provision.fog.smithi136:[Errno None] Unable to connect to port 22 on 172.21.15.136
2023-11-04T03:51:41.628 WARNING:teuthology.provision.fog.smithi083:[Errno None] Unable to connect to port 22 on 172.21.15.83
2023-11-04T03:51:42.426 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:51:46.430 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:51:47.540 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:51:47.541 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T03:51:47.541 DEBUG:teuthology.provision.fog.smithi124:Canceling deploy task with ID 582285
2023-11-04T03:51:47.541 DEBUG:teuthology.provision.fog.smithi124:FOG request: DELETE /task/cancel {"id": 582285}
2023-11-04T03:51:47.598 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi124
2023-11-04T03:51:47.599 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:51:47.628 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:51:47.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:51:47.740 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=56ecef8e1f6b4ff5922cd87cc09d572e
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi124
2023-11-04T03:51:47.742 INFO:teuthology.nuke:Checking targets against current locks
2023-11-04T03:51:47.808 DEBUG:teuthology.nuke:shortname: smithi083
2023-11-04T03:51:47.809 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-04T03:51:47.833 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-04_03:36:46-smoke-reef-release-distro-default-smithi/7446729', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-04 03:47:20.429803', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFz+5Oq0McWZyLQzQnK4RE5dASXkhMEViU1mD3kndldPFXrKEOY/x+u8GheAk1NoXHai2AhpRm++lhbi066cO0o='}
2023-11-04T03:51:47.834 DEBUG:teuthology.nuke:shortname: smithi124
2023-11-04T03:51:47.835 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-04T03:51:47.857 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi124.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-04_03:36:46-smoke-reef-release-distro-default-smithi/7446729', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-04 03:47:20.432903', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}
2023-11-04T03:51:47.862 DEBUG:teuthology.nuke:shortname: smithi136
2023-11-04T03:51:47.863 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-04T03:51:47.884 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-04_03:36:46-smoke-reef-release-distro-default-smithi/7446729', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-04 03:47:20.431576', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMn9bYfRIp/m4qcArm/kvduJGQeVVJZprHKU+dI0En7meDOcBgcyCBvA8TevPhx+EatcfJACLPi2NQ4Rh4NYSuo='}
2023-11-04T03:51:47.892 INFO:teuthology.orchestra.console.smithi083:Power off
2023-11-04T03:51:47.893 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T03:51:47.907 INFO:teuthology.orchestra.console.smithi124:Power off
2023-11-04T03:51:47.909 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T03:51:47.917 DEBUG:teuthology.orchestra.console.smithi083:power off output: Chassis Power Control: Down/Off
2023-11-04T03:51:47.918 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:51:47.938 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on
2023-11-04T03:51:47.955 INFO:teuthology.orchestra.console.smithi136:Power off
2023-11-04T03:51:47.956 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T03:51:47.978 DEBUG:teuthology.orchestra.console.smithi136:power off output: Chassis Power Control: Down/Off
2023-11-04T03:51:47.979 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:51:48.001 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on
2023-11-04T03:51:50.682 WARNING:teuthology.provision.fog.smithi136:[Errno None] Unable to connect to port 22 on 172.21.15.136
2023-11-04T03:51:50.683 WARNING:teuthology.provision.fog.smithi083:[Errno None] Unable to connect to port 22 on 172.21.15.83
2023-11-04T03:51:51.940 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:51:52.002 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:51:52.050 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is off
2023-11-04T03:51:52.113 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on
2023-11-04T03:51:52.152 INFO:teuthology.orchestra.console.smithi083:Power off completed
2023-11-04T03:51:52.306 INFO:teuthology.lock.ops:unlocked: smithi083.front.sepia.ceph.com
2023-11-04T03:51:56.114 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:51:56.224 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on
2023-11-04T03:51:56.685 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:51:56.688 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:51:59.743 WARNING:teuthology.provision.fog.smithi136:[Errno None] Unable to connect to port 22 on 172.21.15.136
2023-11-04T03:51:59.743 WARNING:teuthology.provision.fog.smithi083:[Errno None] Unable to connect to port 22 on 172.21.15.83
2023-11-04T03:52:00.225 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:52:00.335 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is off
2023-11-04T03:52:00.436 INFO:teuthology.orchestra.console.smithi136:Power off completed
2023-11-04T03:52:00.589 INFO:teuthology.lock.ops:unlocked: smithi136.front.sepia.ceph.com
2023-11-04T03:52:05.750 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:52:05.753 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:52:07.981 DEBUG:teuthology.orchestra.console.smithi124:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T03:52:07.981 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T03:52:08.827 WARNING:teuthology.provision.fog.smithi136:[Errno None] Unable to connect to port 22 on 172.21.15.136
2023-11-04T03:52:08.828 WARNING:teuthology.provision.fog.smithi083:[Errno None] Unable to connect to port 22 on 172.21.15.83
2023-11-04T03:52:14.835 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:52:14.837 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:52:17.914 WARNING:teuthology.provision.fog.smithi083:[Errno None] Unable to connect to port 22 on 172.21.15.83
2023-11-04T03:52:17.915 WARNING:teuthology.provision.fog.smithi136:[Errno None] Unable to connect to port 22 on 172.21.15.136
2023-11-04T03:52:23.919 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:52:23.922 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:52:26.970 WARNING:teuthology.provision.fog.smithi083:[Errno None] Unable to connect to port 22 on 172.21.15.83
2023-11-04T03:52:27.006 WARNING:teuthology.provision.fog.smithi136:[Errno None] Unable to connect to port 22 on 172.21.15.136
2023-11-04T03:52:28.052 DEBUG:teuthology.orchestra.console.smithi124:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T03:52:28.053 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T03:52:32.975 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:52:33.008 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:52:36.026 WARNING:teuthology.provision.fog.smithi083:[Errno None] Unable to connect to port 22 on 172.21.15.83
2023-11-04T03:52:36.090 WARNING:teuthology.provision.fog.smithi136:[Errno None] Unable to connect to port 22 on 172.21.15.136
2023-11-04T03:52:42.030 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:52:42.092 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:52:45.082 WARNING:teuthology.provision.fog.smithi083:[Errno None] Unable to connect to port 22 on 172.21.15.83
2023-11-04T03:52:45.146 WARNING:teuthology.provision.fog.smithi136:[Errno None] Unable to connect to port 22 on 172.21.15.136
2023-11-04T03:52:48.124 DEBUG:teuthology.orchestra.console.smithi124:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T03:52:48.125 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:52:49.133 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:52:51.087 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:52:51.148 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:52:53.135 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:52:54.139 WARNING:teuthology.provision.fog.smithi083:[Errno None] Unable to connect to port 22 on 172.21.15.83
2023-11-04T03:52:54.202 WARNING:teuthology.provision.fog.smithi136:[Errno None] Unable to connect to port 22 on 172.21.15.136
2023-11-04T03:52:54.245 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:52:58.246 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:52:59.357 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:53:00.143 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:53:00.204 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:53:03.194 WARNING:teuthology.provision.fog.smithi083:[Errno None] Unable to connect to port 22 on 172.21.15.83
2023-11-04T03:53:03.259 WARNING:teuthology.provision.fog.smithi136:[Errno None] Unable to connect to port 22 on 172.21.15.136
2023-11-04T03:53:03.359 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:53:04.469 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:53:08.474 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:53:09.197 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:53:09.261 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:53:09.585 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:53:12.318 WARNING:teuthology.provision.fog.smithi136:[Errno None] Unable to connect to port 22 on 172.21.15.136
2023-11-04T03:53:13.587 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:53:14.696 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:53:18.323 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:53:18.697 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:53:19.806 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:53:21.406 WARNING:teuthology.provision.fog.smithi136:[Errno None] Unable to connect to port 22 on 172.21.15.136
2023-11-04T03:53:23.807 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:53:24.916 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:53:27.411 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:53:27.678 WARNING:teuthology.provision.fog.smithi083:[Errno None] Unable to connect to port 22 on 172.21.15.83
2023-11-04T03:53:28.918 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:53:30.027 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:53:33.534 WARNING:teuthology.provision.fog.smithi136:[Errno None] Unable to connect to port 22 on 172.21.15.136
2023-11-04T03:53:33.680 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:53:34.030 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:53:35.138 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:53:36.639 WARNING:teuthology.provision.fog.smithi083:[Errno None] Unable to connect to port 22 on 172.21.15.83
2023-11-04T03:53:39.142 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:53:39.537 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:53:40.251 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:53:42.622 WARNING:teuthology.provision.fog.smithi136:[Errno None] Unable to connect to port 22 on 172.21.15.136
2023-11-04T03:53:42.641 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:53:42.766 ERROR:teuthology.orchestra.connection:Error authenticating with smithi083.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2023-11-04T03:53:44.253 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:53:45.363 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:53:46.899 ERROR:teuthology.orchestra.connection:Error authenticating with smithi083.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2023-11-04T03:53:48.624 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:53:48.626 WARNING:teuthology.provision.fog.smithi136:[Errno None] Unable to connect to port 22 on 172.21.15.136
2023-11-04T03:53:49.364 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:53:50.475 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:53:53.990 ERROR:teuthology.orchestra.connection:Error authenticating with smithi083.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2023-11-04T03:53:54.476 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:53:54.629 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T03:53:54.752 ERROR:teuthology.orchestra.connection:Error authenticating with smithi136.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2023-11-04T03:53:55.585 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:53:58.836 ERROR:teuthology.orchestra.connection:Error authenticating with smithi136.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2023-11-04T03:53:59.586 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:54:00.696 DEBUG:teuthology.orchestra.console.smithi124:check power output: 
2023-11-04T03:54:00.697 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T03:54:00.697 ERROR:teuthology.orchestra.console.smithi124:Failed to power off
2023-11-04T03:54:00.879 INFO:teuthology.lock.ops:unlocked: smithi124.front.sepia.ceph.com
2023-11-04T03:54:01.272 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi124