2023-09-21T12:36:54.989 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-21T12:36:54.991 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-09-21T12:36:55.169 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f7191b62d00>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f719140dca0>, signals=[15])
2023-09-21T12:36:55.171 INFO:teuthology.lock.ops:Start node 'smithi082.front.sepia.ceph.com' reimaging
2023-09-21T12:36:55.171 INFO:teuthology.lock.ops:Updating [smithi082.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T12:36:55.171 INFO:teuthology.lock.ops:Updating smithi082.front.sepia.ceph.com on lock server
2023-09-21T12:36:55.200 INFO:teuthology.lock.ops:Start node 'smithi105.front.sepia.ceph.com' reimaging
2023-09-21T12:36:55.201 INFO:teuthology.lock.ops:Updating [smithi105.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T12:36:55.201 INFO:teuthology.lock.ops:Updating smithi105.front.sepia.ceph.com on lock server
2023-09-21T12:36:55.202 DEBUG:teuthology.provision.fog.smithi082:FOG request: GET /host {"name": "smithi082"}
2023-09-21T12:36:55.224 INFO:teuthology.lock.ops:Start node 'smithi186.front.sepia.ceph.com' reimaging
2023-09-21T12:36:55.225 INFO:teuthology.lock.ops:Updating [smithi186.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T12:36:55.225 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server
2023-09-21T12:36:55.226 DEBUG:teuthology.provision.fog.smithi105:FOG request: GET /host {"name": "smithi105"}
2023-09-21T12:36:55.240 DEBUG:teuthology.provision.fog.smithi186:FOG request: GET /host {"name": "smithi186"}
2023-09-21T12:36:55.274 DEBUG:teuthology.provision.fog.smithi082:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-09-21T12:36:55.310 DEBUG:teuthology.provision.fog.smithi105:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-09-21T12:36:55.315 DEBUG:teuthology.provision.fog.smithi186:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-09-21T12:36:55.321 DEBUG:teuthology.provision.fog.smithi082:Requesting image smithi_rhel_8.6 (ID 5)
2023-09-21T12:36:55.322 DEBUG:teuthology.provision.fog.smithi082:FOG request: PUT /host/187 {"imageID": 5}
2023-09-21T12:36:55.352 DEBUG:teuthology.provision.fog.smithi105:Requesting image smithi_rhel_8.6 (ID 5)
2023-09-21T12:36:55.353 DEBUG:teuthology.provision.fog.smithi105:FOG request: PUT /host/210 {"imageID": 5}
2023-09-21T12:36:55.355 DEBUG:teuthology.provision.fog.smithi186:Requesting image smithi_rhel_8.6 (ID 5)
2023-09-21T12:36:55.356 DEBUG:teuthology.provision.fog.smithi186:FOG request: PUT /host/291 {"imageID": 5}
2023-09-21T12:36:55.372 INFO:teuthology.provision.fog.smithi082:Scheduling deploy of rhel 8.6
2023-09-21T12:36:55.372 DEBUG:teuthology.provision.fog.smithi082:FOG request: GET /task/active None
2023-09-21T12:36:55.393 INFO:teuthology.provision.fog.smithi186:Scheduling deploy of rhel 8.6
2023-09-21T12:36:55.393 DEBUG:teuthology.provision.fog.smithi186:FOG request: GET /task/active None
2023-09-21T12:36:55.397 INFO:teuthology.provision.fog.smithi105:Scheduling deploy of rhel 8.6
2023-09-21T12:36:55.397 DEBUG:teuthology.provision.fog.smithi105:FOG request: GET /task/active None
2023-09-21T12:36:55.449 DEBUG:teuthology.provision.fog.smithi082:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T12:36:55.453 DEBUG:teuthology.provision.fog.smithi105:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T12:36:55.469 DEBUG:teuthology.provision.fog.smithi186:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T12:36:55.485 DEBUG:teuthology.provision.fog.smithi105:FOG request: POST /host/210/task {"taskTypeID": 1}
2023-09-21T12:36:55.488 DEBUG:teuthology.provision.fog.smithi082:FOG request: POST /host/187/task {"taskTypeID": 1}
2023-09-21T12:36:55.499 DEBUG:teuthology.provision.fog.smithi186:FOG request: POST /host/291/task {"taskTypeID": 1}
2023-09-21T12:36:55.542 DEBUG:teuthology.provision.fog.smithi105:FOG request: GET /task/active None
2023-09-21T12:36:55.543 DEBUG:teuthology.provision.fog.smithi082:FOG request: GET /task/active None
2023-09-21T12:36:55.557 DEBUG:teuthology.provision.fog.smithi186:FOG request: GET /task/active None
2023-09-21T12:36:55.642 INFO:teuthology.orchestra.console.smithi082:Power off
2023-09-21T12:36:55.643 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:36:55.654 INFO:teuthology.orchestra.console.smithi186:Power off
2023-09-21T12:36:55.655 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:36:55.664 INFO:teuthology.orchestra.console.smithi105:Power off
2023-09-21T12:36:55.665 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:36:55.673 DEBUG:teuthology.orchestra.console.smithi082:power off output: Chassis Power Control: Down/Off
2023-09-21T12:36:55.673 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:36:55.680 DEBUG:teuthology.orchestra.console.smithi186:power off output: Chassis Power Control: Down/Off
2023-09-21T12:36:55.681 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:36:55.692 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on
2023-09-21T12:36:55.704 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on
2023-09-21T12:36:59.694 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:36:59.704 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:36:59.803 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on
2023-09-21T12:36:59.813 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on
2023-09-21T12:37:03.806 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:03.815 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:03.915 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is off
2023-09-21T12:37:03.923 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on
2023-09-21T12:37:04.017 INFO:teuthology.orchestra.console.smithi082:Power off completed
2023-09-21T12:37:04.118 INFO:teuthology.orchestra.console.smithi082:Power on
2023-09-21T12:37:04.118 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T12:37:04.139 DEBUG:teuthology.orchestra.console.smithi082:power on output: Chassis Power Control: Up/On
2023-09-21T12:37:04.139 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:04.159 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is off
2023-09-21T12:37:07.926 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:08.034 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is off
2023-09-21T12:37:08.135 INFO:teuthology.orchestra.console.smithi186:Power off completed
2023-09-21T12:37:08.160 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:08.236 INFO:teuthology.orchestra.console.smithi186:Power on
2023-09-21T12:37:08.237 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T12:37:08.258 DEBUG:teuthology.orchestra.console.smithi186:power on output: Chassis Power Control: Up/On
2023-09-21T12:37:08.258 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:08.269 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is off
2023-09-21T12:37:08.279 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is off
2023-09-21T12:37:12.274 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:12.281 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:12.384 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on
2023-09-21T12:37:12.389 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is off
2023-09-21T12:37:12.485 INFO:teuthology.orchestra.console.smithi082:Power on completed
2023-09-21T12:37:12.586 INFO:teuthology.provision.fog.smithi082:Waiting for deploy to finish
2023-09-21T12:37:12.587 DEBUG:teuthology.provision.fog.smithi082:FOG request: GET /task/active None
2023-09-21T12:37:15.732 DEBUG:teuthology.orchestra.console.smithi105:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:37:15.733 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:37:16.390 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:16.611 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on
2023-09-21T12:37:16.712 INFO:teuthology.orchestra.console.smithi186:Power on completed
2023-09-21T12:37:16.813 INFO:teuthology.provision.fog.smithi186:Waiting for deploy to finish
2023-09-21T12:37:16.813 DEBUG:teuthology.provision.fog.smithi186:FOG request: GET /task/active None
2023-09-21T12:37:27.674 DEBUG:teuthology.provision.fog.smithi082:FOG request: GET /task/active None
2023-09-21T12:37:31.902 DEBUG:teuthology.provision.fog.smithi186:FOG request: GET /task/active None
2023-09-21T12:37:35.805 DEBUG:teuthology.orchestra.console.smithi105:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:37:35.805 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:37:42.766 DEBUG:teuthology.provision.fog.smithi082:FOG request: GET /task/active None
2023-09-21T12:37:46.977 DEBUG:teuthology.provision.fog.smithi186:FOG request: GET /task/active None
2023-09-21T12:37:55.878 DEBUG:teuthology.orchestra.console.smithi105:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:37:55.878 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:56.886 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:37:57.855 DEBUG:teuthology.provision.fog.smithi082:FOG request: GET /task/active None
2023-09-21T12:38:00.890 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:38:01.998 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:38:02.066 DEBUG:teuthology.provision.fog.smithi186:FOG request: GET /task/active None
2023-09-21T12:38:06.003 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:38:07.112 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:38:11.114 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:38:12.222 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:38:12.940 DEBUG:teuthology.provision.fog.smithi082:FOG request: GET /task/active None
2023-09-21T12:38:16.226 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:38:17.139 DEBUG:teuthology.provision.fog.smithi186:FOG request: GET /task/active None
2023-09-21T12:38:17.334 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:38:21.339 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:38:22.447 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:38:26.452 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:38:27.561 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:38:28.023 DEBUG:teuthology.provision.fog.smithi082:FOG request: GET /task/active None
2023-09-21T12:38:31.566 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:38:32.214 DEBUG:teuthology.provision.fog.smithi186:FOG request: GET /task/active None
2023-09-21T12:38:32.674 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:38:36.678 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:38:37.787 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:38:41.790 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:38:42.900 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:38:43.099 DEBUG:teuthology.provision.fog.smithi082:FOG request: GET /task/active None
2023-09-21T12:38:46.903 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:38:47.298 DEBUG:teuthology.provision.fog.smithi186:FOG request: GET /task/active None
2023-09-21T12:38:48.014 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:38:52.018 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:38:53.129 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:38:57.134 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:38:58.185 DEBUG:teuthology.provision.fog.smithi082:FOG request: GET /task/active None
2023-09-21T12:38:58.243 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:39:02.246 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:02.368 DEBUG:teuthology.provision.fog.smithi186:FOG request: GET /task/active None
2023-09-21T12:39:03.356 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:39:07.358 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:08.466 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:39:08.467 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T12:39:08.467 ERROR:teuthology.orchestra.console.smithi105:Failed to power off
2023-09-21T12:39:08.467 INFO:teuthology.orchestra.console.smithi105:Power on
2023-09-21T12:39:08.468 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T12:39:13.292 DEBUG:teuthology.provision.fog.smithi082:FOG request: GET /task/active None
2023-09-21T12:39:17.478 DEBUG:teuthology.provision.fog.smithi186:FOG request: GET /task/active None
2023-09-21T12:39:28.378 DEBUG:teuthology.provision.fog.smithi082:FOG request: GET /task/active None
2023-09-21T12:39:28.541 DEBUG:teuthology.orchestra.console.smithi105:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:39:28.541 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T12:39:32.570 DEBUG:teuthology.provision.fog.smithi186:FOG request: GET /task/active None
2023-09-21T12:39:43.480 DEBUG:teuthology.provision.fog.smithi082:FOG request: GET /task/active None
2023-09-21T12:39:43.555 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:39:47.678 DEBUG:teuthology.provision.fog.smithi186:FOG request: GET /task/active None
2023-09-21T12:39:48.611 DEBUG:teuthology.orchestra.console.smithi105:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:39:48.612 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T12:40:01.950 WARNING:teuthology.provision.fog.smithi082:[Errno None] Unable to connect to port 22 on 172.21.15.82
2023-09-21T12:40:02.788 DEBUG:teuthology.provision.fog.smithi186:FOG request: GET /task/active None
2023-09-21T12:40:02.896 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:40:07.958 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:40:08.679 DEBUG:teuthology.orchestra.console.smithi105:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:40:08.680 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:40:09.688 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:40:11.038 WARNING:teuthology.provision.fog.smithi082:[Errno None] Unable to connect to port 22 on 172.21.15.82
2023-09-21T12:40:13.691 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:40:14.800 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:40:17.042 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:40:18.803 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:40:19.913 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:40:20.126 WARNING:teuthology.provision.fog.smithi082:[Errno None] Unable to connect to port 22 on 172.21.15.82
2023-09-21T12:40:21.406 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186
2023-09-21T12:40:23.917 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:40:25.030 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:40:26.129 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:40:27.410 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:40:29.033 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:40:29.214 WARNING:teuthology.provision.fog.smithi082:[Errno None] Unable to connect to port 22 on 172.21.15.82
2023-09-21T12:40:30.142 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:40:30.490 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186
2023-09-21T12:40:34.146 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:40:35.217 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:40:35.254 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:40:36.493 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:40:38.298 WARNING:teuthology.provision.fog.smithi082:[Errno None] Unable to connect to port 22 on 172.21.15.82
2023-09-21T12:40:39.256 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:40:39.546 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186
2023-09-21T12:40:40.365 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:40:44.302 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:40:44.367 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:40:45.476 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:40:45.548 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:40:48.602 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186
2023-09-21T12:40:49.479 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:40:50.591 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:40:54.594 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:40:54.604 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:40:55.703 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:40:57.658 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186
2023-09-21T12:40:59.706 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:41:00.814 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:41:03.663 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:41:04.816 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:41:05.925 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:41:09.930 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:41:11.039 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:41:15.042 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:41:16.150 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:41:20.154 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:41:21.262 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:41:21.263 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T12:41:21.263 DEBUG:teuthology.provision.fog.smithi105:Canceling deploy task with ID 499567
2023-09-21T12:41:21.263 DEBUG:teuthology.provision.fog.smithi105:FOG request: DELETE /task/cancel {"id": 499567}
2023-09-21T12:41:21.332 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi105
2023-09-21T12:41:21.333 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-09-21T12:41:21.474 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=0030f9989b6f4200a87d4d6273b332b6
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi105
2023-09-21T12:41:21.477 INFO:teuthology.nuke:Checking targets against current locks
2023-09-21T12:41:21.551 DEBUG:teuthology.nuke:shortname: smithi082
2023-09-21T12:41:21.552 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T12:41:21.578 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-09-20_16:32:45-fs-wip-lusov-config-format-distro-default-smithi/7399886', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 12:36:54.081892', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIoRqOxtq/1fwHQy58PWMVsJ0o5UjhJPaY9rkMSykLM29zDOYZ8lTEGZWN+Lmhpi4CACEGe37VlJy87K8xRjcEk='}
2023-09-21T12:41:21.584 DEBUG:teuthology.nuke:shortname: smithi105
2023-09-21T12:41:21.585 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T12:41:21.606 DEBUG:teuthology.nuke:shortname: smithi186
2023-09-21T12:41:21.607 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T12:41:21.608 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-09-20_16:32:45-fs-wip-lusov-config-format-distro-default-smithi/7399886', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 12:36:54.083649', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJptX02Xv1cnuL+0u1sr/6lwByx6nivuBRCcqhvjH1/U0Rf2g+qKaUsMSwf+FRz6zuwx5VHORTRpsfjGGHXK4a8='}
2023-09-21T12:41:21.633 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-09-20_16:32:45-fs-wip-lusov-config-format-distro-default-smithi/7399886', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 12:36:54.087713', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHUqNOjtNjIRPwm8xjvIyo7kPaqkXNz65qeYmgZdsyLYgyfa1gzIpZcrkYyTyPMICylTTySh3/5CHHelkOFmtKM='}
2023-09-21T12:41:21.639 INFO:teuthology.orchestra.console.smithi082:Power off
2023-09-21T12:41:21.640 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:41:21.661 DEBUG:teuthology.orchestra.console.smithi082:power off output: Chassis Power Control: Down/Off
2023-09-21T12:41:21.661 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:41:21.676 INFO:teuthology.orchestra.console.smithi105:Power off
2023-09-21T12:41:21.677 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:41:21.688 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on
2023-09-21T12:41:21.716 INFO:teuthology.orchestra.console.smithi186:Power off
2023-09-21T12:41:21.717 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:41:21.739 DEBUG:teuthology.orchestra.console.smithi186:power off output: Chassis Power Control: Down/Off
2023-09-21T12:41:21.740 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:41:21.762 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on
2023-09-21T12:41:25.690 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:41:25.764 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:41:25.799 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on
2023-09-21T12:41:25.874 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on
2023-09-21T12:41:29.802 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:41:29.875 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:41:29.911 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is off
2023-09-21T12:41:29.985 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is off
2023-09-21T12:41:30.013 INFO:teuthology.orchestra.console.smithi082:Power off completed
2023-09-21T12:41:30.086 INFO:teuthology.orchestra.console.smithi186:Power off completed
2023-09-21T12:41:30.173 INFO:teuthology.lock.ops:unlocked: smithi082.front.sepia.ceph.com
2023-09-21T12:41:30.239 INFO:teuthology.lock.ops:unlocked: smithi186.front.sepia.ceph.com
2023-09-21T12:41:41.744 DEBUG:teuthology.orchestra.console.smithi105:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:41:41.744 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:41:44.306 WARNING:teuthology.provision.fog.smithi082:timed out
2023-09-21T12:41:50.315 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:42:01.813 DEBUG:teuthology.orchestra.console.smithi105:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:42:01.813 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:42:03.666 WARNING:teuthology.provision.fog.smithi186:timed out
2023-09-21T12:42:08.667 WARNING:teuthology.provision.fog.smithi082:[Errno None] Unable to connect to port 22 on 172.21.15.82
2023-09-21T12:42:09.670 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:42:12.730 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186
2023-09-21T12:42:14.670 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:42:17.722 WARNING:teuthology.provision.fog.smithi082:[Errno None] Unable to connect to port 22 on 172.21.15.82
2023-09-21T12:42:18.733 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:42:21.786 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186
2023-09-21T12:42:21.884 DEBUG:teuthology.orchestra.console.smithi105:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:42:21.884 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:42:22.895 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:42:23.725 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:42:26.778 WARNING:teuthology.provision.fog.smithi082:[Errno None] Unable to connect to port 22 on 172.21.15.82
2023-09-21T12:42:26.897 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:42:27.789 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:42:28.009 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:42:30.842 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186
2023-09-21T12:42:32.011 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:42:32.781 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:42:33.123 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:42:35.838 WARNING:teuthology.provision.fog.smithi082:[Errno None] Unable to connect to port 22 on 172.21.15.82
2023-09-21T12:42:36.845 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:42:37.125 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:42:38.236 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:42:39.902 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186
2023-09-21T12:42:41.841 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:42:42.239 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:42:43.352 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:42:44.926 WARNING:teuthology.provision.fog.smithi082:[Errno None] Unable to connect to port 22 on 172.21.15.82
2023-09-21T12:42:45.905 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:42:47.355 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:42:48.466 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:42:48.986 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186
2023-09-21T12:42:50.930 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:42:52.469 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:42:53.578 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:42:54.989 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:42:57.582 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:42:58.042 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186
2023-09-21T12:42:58.693 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:43:02.694 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:43:03.803 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:43:04.045 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:43:07.808 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:43:08.918 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:43:09.338 WARNING:teuthology.provision.fog.smithi082:[Errno None] Unable to connect to port 22 on 172.21.15.82
2023-09-21T12:43:12.922 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:43:14.032 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:43:14.202 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186
2023-09-21T12:43:15.342 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:43:18.036 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:43:18.398 WARNING:teuthology.provision.fog.smithi082:[Errno None] Unable to connect to port 22 on 172.21.15.82
2023-09-21T12:43:19.149 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:43:20.205 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:43:23.154 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:43:23.259 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186
2023-09-21T12:43:24.302 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:43:24.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:43:24.526 ERROR:teuthology.orchestra.connection:Error authenticating with smithi082.front.sepia.ceph.com: Authentication failed.
2023-09-21T12:43:24.530 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-09-21T12:43:24.530 DEBUG:teuthology.orchestra.run.smithi082:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-09-21T12:43:24.533 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:43:25.540 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:43:26.546 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:43:27.552 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:43:28.303 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:43:28.560 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:43:29.261 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:43:29.383 ERROR:teuthology.orchestra.connection:Error authenticating with smithi186.front.sepia.ceph.com: Authentication failed.
2023-09-21T12:43:29.386 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-09-21T12:43:29.387 DEBUG:teuthology.orchestra.run.smithi186:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-09-21T12:43:29.389 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:43:29.413 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:43:29.570 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:43:30.398 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:43:30.580 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:43:31.409 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:43:31.589 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:43:32.420 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:43:32.599 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:43:33.414 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:43:33.431 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:43:33.608 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:43:34.442 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:43:34.524 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-09-21T12:43:34.525 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T12:43:34.525 ERROR:teuthology.orchestra.console.smithi105:Failed to power off
2023-09-21T12:43:34.619 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:43:34.675 INFO:teuthology.lock.ops:unlocked: smithi105.front.sepia.ceph.com
2023-09-21T12:43:35.001 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi105