2023-09-21T14:28:59.402 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-21T14:28:59.403 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-09-21T14:28:59.617 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7fe8fcf28d00>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7fe8fd9dc670>, signals=[15])
2023-09-21T14:28:59.619 INFO:teuthology.lock.ops:Start node 'smithi016.front.sepia.ceph.com' reimaging
2023-09-21T14:28:59.619 INFO:teuthology.lock.ops:Updating [smithi016.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T14:28:59.620 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server
2023-09-21T14:28:59.645 INFO:teuthology.lock.ops:Start node 'smithi090.front.sepia.ceph.com' reimaging
2023-09-21T14:28:59.646 INFO:teuthology.lock.ops:Updating [smithi090.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T14:28:59.646 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server
2023-09-21T14:28:59.647 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /host {"name": "smithi016"}
2023-09-21T14:28:59.669 INFO:teuthology.lock.ops:Start node 'smithi107.front.sepia.ceph.com' reimaging
2023-09-21T14:28:59.670 INFO:teuthology.lock.ops:Updating [smithi107.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T14:28:59.670 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server
2023-09-21T14:28:59.671 DEBUG:teuthology.provision.fog.smithi090:FOG request: GET /host {"name": "smithi090"}
2023-09-21T14:28:59.694 INFO:teuthology.lock.ops:Start node 'smithi112.front.sepia.ceph.com' reimaging
2023-09-21T14:28:59.695 INFO:teuthology.lock.ops:Updating [smithi112.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T14:28:59.695 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server
2023-09-21T14:28:59.696 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /host {"name": "smithi107"}
2023-09-21T14:28:59.719 DEBUG:teuthology.provision.fog.smithi112:FOG request: GET /host {"name": "smithi112"}
2023-09-21T14:28:59.727 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /image {"name": "smithi_ubuntu_22.04"}
2023-09-21T14:28:59.740 DEBUG:teuthology.provision.fog.smithi090:FOG request: GET /image {"name": "smithi_ubuntu_22.04"}
2023-09-21T14:28:59.771 DEBUG:teuthology.provision.fog.smithi016:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-09-21T14:28:59.771 DEBUG:teuthology.provision.fog.smithi016:FOG request: PUT /host/122 {"imageID": 6}
2023-09-21T14:28:59.779 DEBUG:teuthology.provision.fog.smithi090:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-09-21T14:28:59.779 DEBUG:teuthology.provision.fog.smithi090:FOG request: PUT /host/195 {"imageID": 6}
2023-09-21T14:28:59.785 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /image {"name": "smithi_ubuntu_22.04"}
2023-09-21T14:28:59.788 DEBUG:teuthology.provision.fog.smithi112:FOG request: GET /image {"name": "smithi_ubuntu_22.04"}
2023-09-21T14:28:59.822 INFO:teuthology.provision.fog.smithi016:Scheduling deploy of ubuntu 22.04
2023-09-21T14:28:59.823 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /task/active None
2023-09-21T14:28:59.827 DEBUG:teuthology.provision.fog.smithi112:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-09-21T14:28:59.827 DEBUG:teuthology.provision.fog.smithi112:FOG request: PUT /host/217 {"imageID": 6}
2023-09-21T14:28:59.836 DEBUG:teuthology.provision.fog.smithi107:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-09-21T14:28:59.836 DEBUG:teuthology.provision.fog.smithi107:FOG request: PUT /host/212 {"imageID": 6}
2023-09-21T14:28:59.839 INFO:teuthology.provision.fog.smithi090:Scheduling deploy of ubuntu 22.04
2023-09-21T14:28:59.840 DEBUG:teuthology.provision.fog.smithi090:FOG request: GET /task/active None
2023-09-21T14:28:59.881 INFO:teuthology.provision.fog.smithi107:Scheduling deploy of ubuntu 22.04
2023-09-21T14:28:59.881 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /task/active None
2023-09-21T14:28:59.889 INFO:teuthology.provision.fog.smithi112:Scheduling deploy of ubuntu 22.04
2023-09-21T14:28:59.890 DEBUG:teuthology.provision.fog.smithi112:FOG request: GET /task/active None
2023-09-21T14:28:59.902 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T14:28:59.925 DEBUG:teuthology.provision.fog.smithi090:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T14:28:59.937 DEBUG:teuthology.provision.fog.smithi016:FOG request: POST /host/122/task {"taskTypeID": 1}
2023-09-21T14:28:59.941 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T14:28:59.959 DEBUG:teuthology.provision.fog.smithi112:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T14:28:59.962 DEBUG:teuthology.provision.fog.smithi090:FOG request: POST /host/195/task {"taskTypeID": 1}
2023-09-21T14:28:59.977 DEBUG:teuthology.provision.fog.smithi107:FOG request: POST /host/212/task {"taskTypeID": 1}
2023-09-21T14:29:00.002 DEBUG:teuthology.provision.fog.smithi112:FOG request: POST /host/217/task {"taskTypeID": 1}
2023-09-21T14:29:00.011 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /task/active None
2023-09-21T14:29:00.022 DEBUG:teuthology.provision.fog.smithi090:FOG request: GET /task/active None
2023-09-21T14:29:00.039 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /task/active None
2023-09-21T14:29:00.057 DEBUG:teuthology.provision.fog.smithi112:FOG request: GET /task/active None
2023-09-21T14:29:00.129 INFO:teuthology.orchestra.console.smithi107:Power off
2023-09-21T14:29:00.130 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T14:29:00.147 INFO:teuthology.orchestra.console.smithi090:Power off
2023-09-21T14:29:00.147 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T14:29:00.158 INFO:teuthology.orchestra.console.smithi016:Power off
2023-09-21T14:29:00.159 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T14:29:00.169 DEBUG:teuthology.orchestra.console.smithi090:power off output: Chassis Power Control: Down/Off
2023-09-21T14:29:00.169 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:29:00.182 DEBUG:teuthology.orchestra.console.smithi016:power off output: Chassis Power Control: Down/Off
2023-09-21T14:29:00.183 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:29:00.194 INFO:teuthology.orchestra.console.smithi112:Power off
2023-09-21T14:29:00.195 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T14:29:00.202 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on
2023-09-21T14:29:00.203 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is off
2023-09-21T14:29:00.304 INFO:teuthology.orchestra.console.smithi016:Power off completed
2023-09-21T14:29:00.404 INFO:teuthology.orchestra.console.smithi016:Power on
2023-09-21T14:29:00.405 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T14:29:00.429 DEBUG:teuthology.orchestra.console.smithi016:power on output: Chassis Power Control: Up/On
2023-09-21T14:29:00.430 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:29:00.449 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is off
2023-09-21T14:29:04.206 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:29:04.314 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on
2023-09-21T14:29:04.451 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:29:04.559 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is off
2023-09-21T14:29:08.320 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:29:08.428 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is off
2023-09-21T14:29:08.529 INFO:teuthology.orchestra.console.smithi090:Power off completed
2023-09-21T14:29:08.560 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:29:08.630 INFO:teuthology.orchestra.console.smithi090:Power on
2023-09-21T14:29:08.631 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T14:29:08.650 DEBUG:teuthology.orchestra.console.smithi090:power on output: Chassis Power Control: Up/On
2023-09-21T14:29:08.650 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:29:08.668 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is on
2023-09-21T14:29:08.669 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is off
2023-09-21T14:29:08.769 INFO:teuthology.orchestra.console.smithi016:Power on completed
2023-09-21T14:29:08.870 INFO:teuthology.provision.fog.smithi016:Waiting for deploy to finish
2023-09-21T14:29:08.870 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /task/active None
2023-09-21T14:29:12.674 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:29:12.782 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is off
2023-09-21T14:29:16.787 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:29:16.980 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on
2023-09-21T14:29:17.081 INFO:teuthology.orchestra.console.smithi090:Power on completed
2023-09-21T14:29:17.182 INFO:teuthology.provision.fog.smithi090:Waiting for deploy to finish
2023-09-21T14:29:17.182 DEBUG:teuthology.provision.fog.smithi090:FOG request: GET /task/active None
2023-09-21T14:29:20.196 DEBUG:teuthology.orchestra.console.smithi107:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T14:29:20.196 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T14:29:20.262 DEBUG:teuthology.orchestra.console.smithi112:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T14:29:20.263 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T14:29:23.965 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /task/active None
2023-09-21T14:29:32.275 DEBUG:teuthology.provision.fog.smithi090:FOG request: GET /task/active None
2023-09-21T14:29:39.044 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /task/active None
2023-09-21T14:29:40.263 DEBUG:teuthology.orchestra.console.smithi107:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T14:29:40.264 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T14:29:40.329 DEBUG:teuthology.orchestra.console.smithi112:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T14:29:40.329 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T14:29:47.358 DEBUG:teuthology.provision.fog.smithi090:FOG request: GET /task/active None
2023-09-21T14:29:54.134 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /task/active None
2023-09-21T14:30:00.334 DEBUG:teuthology.orchestra.console.smithi107:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T14:30:00.335 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:30:00.395 DEBUG:teuthology.orchestra.console.smithi112:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T14:30:00.396 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:30:01.342 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:30:01.405 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:30:02.445 DEBUG:teuthology.provision.fog.smithi090:FOG request: GET /task/active None
2023-09-21T14:30:05.346 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:30:05.406 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:30:06.456 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:30:06.515 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:30:09.225 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /task/active None
2023-09-21T14:30:10.459 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:30:10.517 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:30:11.568 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:30:11.628 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:30:15.574 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:30:15.630 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:30:16.684 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:30:16.740 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:30:17.521 DEBUG:teuthology.provision.fog.smithi090:FOG request: GET /task/active None
2023-09-21T14:30:20.688 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:30:20.741 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:30:21.798 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:30:21.851 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:30:24.302 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /task/active None
2023-09-21T14:30:25.800 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:30:25.852 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:30:26.910 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:30:26.961 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:30:30.914 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:30:30.962 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:30:32.025 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:30:32.071 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:30:32.604 DEBUG:teuthology.provision.fog.smithi090:FOG request: GET /task/active None
2023-09-21T14:30:36.029 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:30:36.072 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:30:37.139 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:30:37.180 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:30:39.370 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /task/active None
2023-09-21T14:30:41.142 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:30:41.182 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:30:42.253 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:30:42.290 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:30:46.258 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:30:46.291 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:30:47.367 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:30:47.401 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:30:47.707 DEBUG:teuthology.provision.fog.smithi090:FOG request: GET /task/active None
2023-09-21T14:30:51.370 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:30:51.402 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:30:52.481 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:30:52.511 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:30:54.442 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /task/active None
2023-09-21T14:30:56.484 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:30:56.512 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:30:57.595 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:30:57.621 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:31:01.601 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:31:01.622 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:31:02.712 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:31:02.733 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:31:02.780 DEBUG:teuthology.provision.fog.smithi090:FOG request: GET /task/active None
2023-09-21T14:31:06.716 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:31:06.734 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:31:07.825 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:31:07.845 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:31:09.528 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /task/active None
2023-09-21T14:31:11.829 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:31:11.846 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:31:12.939 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:31:12.940 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T14:31:12.941 ERROR:teuthology.orchestra.console.smithi107:Failed to power off
2023-09-21T14:31:12.941 INFO:teuthology.orchestra.console.smithi107:Power on
2023-09-21T14:31:12.941 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T14:31:12.955 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:31:12.956 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T14:31:12.956 ERROR:teuthology.orchestra.console.smithi112:Failed to power off
2023-09-21T14:31:12.956 INFO:teuthology.orchestra.console.smithi112:Power on
2023-09-21T14:31:12.957 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T14:31:17.866 DEBUG:teuthology.provision.fog.smithi090:FOG request: GET /task/active None
2023-09-21T14:31:24.614 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /task/active None
2023-09-21T14:31:32.954 DEBUG:teuthology.provision.fog.smithi090:FOG request: GET /task/active None
2023-09-21T14:31:33.012 DEBUG:teuthology.orchestra.console.smithi107:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T14:31:33.012 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T14:31:33.024 DEBUG:teuthology.orchestra.console.smithi112:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T14:31:33.024 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T14:31:39.702 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /task/active None
2023-09-21T14:31:48.050 DEBUG:teuthology.provision.fog.smithi090:FOG request: GET /task/active None
2023-09-21T14:31:53.085 DEBUG:teuthology.orchestra.console.smithi107:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T14:31:53.086 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T14:31:53.093 DEBUG:teuthology.orchestra.console.smithi112:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T14:31:53.094 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T14:31:54.801 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /task/active None
2023-09-21T14:32:03.156 DEBUG:teuthology.provision.fog.smithi090:FOG request: GET /task/active None
2023-09-21T14:32:09.890 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /task/active None
2023-09-21T14:32:09.980 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:32:13.155 DEBUG:teuthology.orchestra.console.smithi107:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T14:32:13.156 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:32:13.166 DEBUG:teuthology.orchestra.console.smithi112:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T14:32:13.166 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:32:14.166 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:32:14.173 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:32:18.170 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:32:18.177 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:32:18.234 DEBUG:teuthology.provision.fog.smithi090:FOG request: GET /task/active None
2023-09-21T14:32:19.279 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:32:19.285 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:32:23.282 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:32:23.289 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:32:24.391 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:32:24.397 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:32:28.314 WARNING:teuthology.provision.fog.smithi016:[Errno None] Unable to connect to port 22 on 172.21.15.16
2023-09-21T14:32:28.392 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:32:28.400 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:32:29.504 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:32:29.508 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:32:33.326 DEBUG:teuthology.provision.fog.smithi090:FOG request: GET /task/active None
2023-09-21T14:32:33.505 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:32:33.512 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:32:34.317 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:32:34.614 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:32:34.619 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:32:37.370 WARNING:teuthology.provision.fog.smithi016:[Errno None] Unable to connect to port 22 on 172.21.15.16
2023-09-21T14:32:38.616 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:32:38.623 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:32:39.726 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:32:39.731 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:32:43.374 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:32:43.728 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:32:43.735 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:32:44.838 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:32:44.845 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:32:46.426 WARNING:teuthology.provision.fog.smithi016:[Errno None] Unable to connect to port 22 on 172.21.15.16
2023-09-21T14:32:48.412 DEBUG:teuthology.provision.fog.smithi090:FOG request: GET /task/active None
2023-09-21T14:32:48.839 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:32:48.848 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:32:49.950 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:32:49.957 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:32:52.431 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:32:53.952 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:32:53.959 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:32:55.062 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:32:55.066 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:32:55.486 WARNING:teuthology.provision.fog.smithi016:[Errno None] Unable to connect to port 22 on 172.21.15.16
2023-09-21T14:32:59.066 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:32:59.073 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:33:00.176 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:33:00.181 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:33:01.489 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:33:03.518 DEBUG:teuthology.provision.fog.smithi090:FOG request: GET /task/active None
2023-09-21T14:33:03.608 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:33:04.178 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:33:04.185 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:33:05.288 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:33:05.297 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:33:09.290 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:33:09.298 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:33:10.400 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:33:10.406 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:33:14.406 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:33:14.415 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:33:15.518 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:33:15.523 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:33:19.522 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:33:19.529 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:33:20.632 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:33:20.640 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:33:22.074 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-09-21T14:33:24.634 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:33:24.642 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:33:25.744 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:33:25.745 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T14:33:25.745 DEBUG:teuthology.provision.fog.smithi107:Canceling deploy task with ID 500029
2023-09-21T14:33:25.746 DEBUG:teuthology.provision.fog.smithi107:FOG request: DELETE /task/cancel {"id": 500029}
2023-09-21T14:33:25.752 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:33:25.752 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T14:33:25.752 DEBUG:teuthology.provision.fog.smithi112:Canceling deploy task with ID 500030
2023-09-21T14:33:25.753 DEBUG:teuthology.provision.fog.smithi112:FOG request: DELETE /task/cancel {"id": 500030}
2023-09-21T14:33:25.797 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi107
2023-09-21T14:33:25.797 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-09-21T14:33:25.948 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e40b42083f714b5ea7ec9d211b20530d
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi107
2023-09-21T14:33:25.951 INFO:teuthology.nuke:Checking targets against current locks
2023-09-21T14:33:26.018 DEBUG:teuthology.nuke:shortname: smithi016
2023-09-21T14:33:26.018 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T14:33:26.040 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/svelar-2023-09-20_07:59:56-rados-wip-svelar-testing-1-2023-09-20-1125-distro-default-smithi/7398817', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 14:28:58.565323', 'locked_by': 'scheduled_svelar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD0Vlr+nAuBv+dokKf+5zTsFliuvNFMwdZ/SgLaCDya45hOQKxfnmVplxS84Ko+ZXzDTQVNCh4rspCeDLSIpyFg='}
2023-09-21T14:33:26.041 DEBUG:teuthology.nuke:shortname: smithi090
2023-09-21T14:33:26.042 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T14:33:26.063 DEBUG:teuthology.nuke:shortname: smithi107
2023-09-21T14:33:26.063 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T14:33:26.068 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/svelar-2023-09-20_07:59:56-rados-wip-svelar-testing-1-2023-09-20-1125-distro-default-smithi/7398817', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 14:28:58.563310', 'locked_by': 'scheduled_svelar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}
2023-09-21T14:33:26.073 DEBUG:teuthology.nuke:shortname: smithi112
2023-09-21T14:33:26.074 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T14:33:26.088 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/svelar-2023-09-20_07:59:56-rados-wip-svelar-testing-1-2023-09-20-1125-distro-default-smithi/7398817', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 14:28:58.567713', 'locked_by': 'scheduled_svelar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}
2023-09-21T14:33:26.092 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/svelar-2023-09-20_07:59:56-rados-wip-svelar-testing-1-2023-09-20-1125-distro-default-smithi/7398817', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 14:28:58.566516', 'locked_by': 'scheduled_svelar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNsN1SFtSMFYxyYNcbgvW1eDZSesLM9jY4HJEMXKIWGOAdf0ieWEX3W/j2HEYjhwPyOGTJiDQ1tGjXWJ9ToC5jc='}
2023-09-21T14:33:26.099 INFO:teuthology.orchestra.console.smithi016:Power off
2023-09-21T14:33:26.100 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T14:33:26.125 DEBUG:teuthology.orchestra.console.smithi016:power off output: Chassis Power Control: Down/Off
2023-09-21T14:33:26.126 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:33:26.146 INFO:teuthology.orchestra.console.smithi090:Power off
2023-09-21T14:33:26.147 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T14:33:26.160 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is on
2023-09-21T14:33:26.171 DEBUG:teuthology.orchestra.console.smithi090:power off output: Chassis Power Control: Down/Off
2023-09-21T14:33:26.171 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:33:26.187 INFO:teuthology.orchestra.console.smithi107:Power off
2023-09-21T14:33:26.187 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T14:33:26.204 INFO:teuthology.orchestra.console.smithi112:Power off
2023-09-21T14:33:26.205 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T14:33:26.215 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on
2023-09-21T14:33:28.078 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:33:30.162 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:33:30.216 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:33:30.271 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is on
2023-09-21T14:33:30.327 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is off
2023-09-21T14:33:30.427 INFO:teuthology.orchestra.console.smithi090:Power off completed
2023-09-21T14:33:30.588 INFO:teuthology.lock.ops:unlocked: smithi090.front.sepia.ceph.com
2023-09-21T14:33:31.130 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-09-21T14:33:34.275 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:33:34.384 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is off
2023-09-21T14:33:34.485 INFO:teuthology.orchestra.console.smithi016:Power off completed
2023-09-21T14:33:34.637 INFO:teuthology.lock.ops:unlocked: smithi016.front.sepia.ceph.com
2023-09-21T14:33:37.134 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:33:40.186 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-09-21T14:33:46.193 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:33:46.263 DEBUG:teuthology.orchestra.console.smithi107:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T14:33:46.264 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T14:33:46.277 DEBUG:teuthology.orchestra.console.smithi112:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T14:33:46.278 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T14:33:49.274 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-09-21T14:33:55.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:33:58.330 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-09-21T14:34:01.494 WARNING:teuthology.provision.fog.smithi016:timed out
2023-09-21T14:34:04.334 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:34:06.335 DEBUG:teuthology.orchestra.console.smithi107:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T14:34:06.336 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T14:34:06.352 DEBUG:teuthology.orchestra.console.smithi112:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T14:34:06.352 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T14:34:07.390 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-09-21T14:34:07.497 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:34:13.394 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:34:16.474 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-09-21T14:34:22.481 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:34:25.562 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-09-21T14:34:25.818 WARNING:teuthology.provision.fog.smithi016:[Errno None] Unable to connect to port 22 on 172.21.15.16
2023-09-21T14:34:26.404 DEBUG:teuthology.orchestra.console.smithi107:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T14:34:26.404 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:34:26.422 DEBUG:teuthology.orchestra.console.smithi112:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T14:34:26.423 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:34:27.414 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:34:27.431 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:34:31.415 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:34:31.433 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:34:31.564 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:34:31.820 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:34:32.526 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:34:32.544 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:34:34.618 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-09-21T14:34:34.874 WARNING:teuthology.provision.fog.smithi016:[Errno None] Unable to connect to port 22 on 172.21.15.16
2023-09-21T14:34:36.528 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:34:36.544 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:34:37.643 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:34:37.653 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:34:40.622 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:34:40.876 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:34:41.646 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:34:41.654 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:34:42.756 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:34:42.762 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:34:43.678 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-09-21T14:34:46.761 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:34:46.769 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:34:47.872 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:34:47.880 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:34:49.682 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:34:51.874 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:34:51.885 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:34:52.762 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-09-21T14:34:52.987 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:34:52.995 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:34:56.990 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:34:57.000 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:34:58.103 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:34:58.110 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:34:58.766 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:35:01.818 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-09-21T14:35:02.104 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:35:02.111 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:35:03.215 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:35:03.220 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:35:07.221 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:35:07.229 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:35:07.821 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:35:08.332 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:35:08.337 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:35:12.336 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:35:12.345 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:35:13.447 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:35:13.454 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:35:17.450 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:35:17.459 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:35:18.562 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:35:18.569 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:35:22.566 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:35:22.574 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:35:23.677 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:35:23.683 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:35:26.234 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-09-21T14:35:27.680 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:35:27.688 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:35:28.790 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:35:28.798 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:35:32.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:35:32.331 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-09-21T14:35:32.792 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:35:32.800 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:35:33.904 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:35:33.911 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:35:37.906 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:35:37.914 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:35:38.334 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:35:38.455 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: Authentication failed.
2023-09-21T14:35:38.460 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-09-21T14:35:38.461 DEBUG:teuthology.orchestra.run.smithi090:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-09-21T14:35:38.464 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:35:39.016 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-09-21T14:35:39.017 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T14:35:39.017 ERROR:teuthology.orchestra.console.smithi107:Failed to power off
2023-09-21T14:35:39.023 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T14:35:39.024 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T14:35:39.024 ERROR:teuthology.orchestra.console.smithi112:Failed to power off
2023-09-21T14:35:39.176 INFO:teuthology.lock.ops:unlocked: smithi107.front.sepia.ceph.com
2023-09-21T14:35:39.193 INFO:teuthology.lock.ops:unlocked: smithi112.front.sepia.ceph.com
2023-09-21T14:35:39.473 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:35:39.621 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi107