2023-11-04T03:57:17.963 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2
2023-11-04T03:57:17.965 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T03:57:18.139 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f1ca5b8da90>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f1ca6692670>, signals=[15])
2023-11-04T03:57:18.140 INFO:teuthology.lock.ops:Start node 'smithi081.front.sepia.ceph.com' reimaging
2023-11-04T03:57:18.141 INFO:teuthology.lock.ops:Updating [smithi081.front.sepia.ceph.com]: reset os type and version on server
2023-11-04T03:57:18.141 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server
2023-11-04T03:57:18.166 INFO:teuthology.lock.ops:Start node 'smithi099.front.sepia.ceph.com' reimaging
2023-11-04T03:57:18.166 INFO:teuthology.lock.ops:Updating [smithi099.front.sepia.ceph.com]: reset os type and version on server
2023-11-04T03:57:18.166 INFO:teuthology.lock.ops:Updating smithi099.front.sepia.ceph.com on lock server
2023-11-04T03:57:18.168 DEBUG:teuthology.provision.fog.smithi081:FOG request: GET /host {"name": "smithi081"}
2023-11-04T03:57:18.191 INFO:teuthology.lock.ops:Start node 'smithi164.front.sepia.ceph.com' reimaging
2023-11-04T03:57:18.191 INFO:teuthology.lock.ops:Updating [smithi164.front.sepia.ceph.com]: reset os type and version on server
2023-11-04T03:57:18.191 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server
2023-11-04T03:57:18.192 DEBUG:teuthology.provision.fog.smithi099:FOG request: GET /host {"name": "smithi099"}
2023-11-04T03:57:18.209 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /host {"name": "smithi164"}
2023-11-04T03:57:18.230 DEBUG:teuthology.provision.fog.smithi081:FOG request: GET /image {"name": "smithi_centos_9.stream"}
2023-11-04T03:57:18.266 DEBUG:teuthology.provision.fog.smithi081:Requesting image smithi_centos_9.stream (ID 10)
2023-11-04T03:57:18.266 DEBUG:teuthology.provision.fog.smithi081:FOG request: PUT /host/186 {"imageID": 10}
2023-11-04T03:57:18.269 DEBUG:teuthology.provision.fog.smithi099:FOG request: GET /image {"name": "smithi_centos_9.stream"}
2023-11-04T03:57:18.276 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /image {"name": "smithi_centos_9.stream"}
2023-11-04T03:57:18.313 DEBUG:teuthology.provision.fog.smithi164:Requesting image smithi_centos_9.stream (ID 10)
2023-11-04T03:57:18.313 DEBUG:teuthology.provision.fog.smithi164:FOG request: PUT /host/269 {"imageID": 10}
2023-11-04T03:57:18.316 INFO:teuthology.provision.fog.smithi081:Scheduling deploy of centos 9.stream
2023-11-04T03:57:18.317 DEBUG:teuthology.provision.fog.smithi081:FOG request: GET /task/active None
2023-11-04T03:57:18.321 DEBUG:teuthology.provision.fog.smithi099:Requesting image smithi_centos_9.stream (ID 10)
2023-11-04T03:57:18.321 DEBUG:teuthology.provision.fog.smithi099:FOG request: PUT /host/204 {"imageID": 10}
2023-11-04T03:57:18.363 INFO:teuthology.provision.fog.smithi164:Scheduling deploy of centos 9.stream
2023-11-04T03:57:18.363 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None
2023-11-04T03:57:18.366 INFO:teuthology.provision.fog.smithi099:Scheduling deploy of centos 9.stream
2023-11-04T03:57:18.366 DEBUG:teuthology.provision.fog.smithi099:FOG request: GET /task/active None
2023-11-04T03:57:18.393 DEBUG:teuthology.provision.fog.smithi081:FOG request: GET /tasktype {"name": "deploy"}
2023-11-04T03:57:18.426 DEBUG:teuthology.provision.fog.smithi099:FOG request: GET /tasktype {"name": "deploy"}
2023-11-04T03:57:18.439 DEBUG:teuthology.provision.fog.smithi081:FOG request: POST /host/186/task {"taskTypeID": 1}
2023-11-04T03:57:18.446 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /tasktype {"name": "deploy"}
2023-11-04T03:57:18.466 DEBUG:teuthology.provision.fog.smithi099:FOG request: POST /host/204/task {"taskTypeID": 1}
2023-11-04T03:57:18.484 DEBUG:teuthology.provision.fog.smithi164:FOG request: POST /host/269/task {"taskTypeID": 1}
2023-11-04T03:57:18.491 DEBUG:teuthology.provision.fog.smithi081:FOG request: GET /task/active None
2023-11-04T03:57:18.511 DEBUG:teuthology.provision.fog.smithi099:FOG request: GET /task/active None
2023-11-04T03:57:18.532 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None
2023-11-04T03:57:18.591 INFO:teuthology.orchestra.console.smithi081:Power off
2023-11-04T03:57:18.592 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T03:57:18.609 INFO:teuthology.orchestra.console.smithi099:Power off
2023-11-04T03:57:18.609 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T03:57:18.619 DEBUG:teuthology.orchestra.console.smithi081:power off output: Chassis Power Control: Down/Off
2023-11-04T03:57:18.620 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:57:18.639 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is off
2023-11-04T03:57:18.645 INFO:teuthology.orchestra.console.smithi164:Power off
2023-11-04T03:57:18.646 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T03:57:18.665 DEBUG:teuthology.orchestra.console.smithi164:power off output: Chassis Power Control: Down/Off
2023-11-04T03:57:18.666 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:57:18.685 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is off
2023-11-04T03:57:18.740 INFO:teuthology.orchestra.console.smithi081:Power off completed
2023-11-04T03:57:18.787 INFO:teuthology.orchestra.console.smithi164:Power off completed
2023-11-04T03:57:18.842 INFO:teuthology.orchestra.console.smithi081:Power on
2023-11-04T03:57:18.842 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T03:57:18.864 DEBUG:teuthology.orchestra.console.smithi081:power on output: Chassis Power Control: Up/On
2023-11-04T03:57:18.865 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:57:18.887 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is off
2023-11-04T03:57:18.888 INFO:teuthology.orchestra.console.smithi164:Power on
2023-11-04T03:57:18.889 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T03:57:18.911 DEBUG:teuthology.orchestra.console.smithi164:power on output: Chassis Power Control: Up/On
2023-11-04T03:57:18.911 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:57:18.936 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is off
2023-11-04T03:57:22.892 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:57:22.937 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:57:23.002 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is off
2023-11-04T03:57:23.047 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is off
2023-11-04T03:57:27.008 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:57:27.048 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:57:27.157 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is on
2023-11-04T03:57:27.187 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on
2023-11-04T03:57:27.258 INFO:teuthology.orchestra.console.smithi164:Power on completed
2023-11-04T03:57:27.288 INFO:teuthology.orchestra.console.smithi081:Power on completed
2023-11-04T03:57:27.360 INFO:teuthology.provision.fog.smithi164:Waiting for deploy to finish
2023-11-04T03:57:27.360 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None
2023-11-04T03:57:27.389 INFO:teuthology.provision.fog.smithi081:Waiting for deploy to finish
2023-11-04T03:57:27.389 DEBUG:teuthology.provision.fog.smithi081:FOG request: GET /task/active None
2023-11-04T03:57:38.675 DEBUG:teuthology.orchestra.console.smithi099:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T03:57:38.675 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T03:57:42.447 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None
2023-11-04T03:57:42.464 DEBUG:teuthology.provision.fog.smithi081:FOG request: GET /task/active None
2023-11-04T03:57:57.538 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None
2023-11-04T03:57:57.539 DEBUG:teuthology.provision.fog.smithi081:FOG request: GET /task/active None
2023-11-04T03:57:58.744 DEBUG:teuthology.orchestra.console.smithi099:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T03:57:58.744 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T03:58:12.630 DEBUG:teuthology.provision.fog.smithi081:FOG request: GET /task/active None
2023-11-04T03:58:12.632 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None
2023-11-04T03:58:18.816 DEBUG:teuthology.orchestra.console.smithi099:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T03:58:18.816 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:58:19.824 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T03:58:23.826 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:58:24.937 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T03:58:27.714 DEBUG:teuthology.provision.fog.smithi081:FOG request: GET /task/active None
2023-11-04T03:58:27.727 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None
2023-11-04T03:58:28.939 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:58:30.048 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T03:58:34.052 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:58:35.160 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T03:58:39.164 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:58:40.277 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T03:58:42.780 DEBUG:teuthology.provision.fog.smithi081:FOG request: GET /task/active None
2023-11-04T03:58:42.793 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None
2023-11-04T03:58:44.280 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:58:45.388 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T03:58:49.390 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:58:50.499 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T03:58:54.502 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:58:55.611 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T03:58:57.862 DEBUG:teuthology.provision.fog.smithi081:FOG request: GET /task/active None
2023-11-04T03:58:57.871 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None
2023-11-04T03:58:59.613 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:59:00.722 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T03:59:04.727 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:59:05.836 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T03:59:09.841 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:59:10.949 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T03:59:12.941 DEBUG:teuthology.provision.fog.smithi081:FOG request: GET /task/active None
2023-11-04T03:59:12.947 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None
2023-11-04T03:59:14.952 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:59:16.061 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T03:59:20.066 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:59:21.174 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T03:59:25.178 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:59:26.286 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T03:59:28.038 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None
2023-11-04T03:59:28.040 DEBUG:teuthology.provision.fog.smithi081:FOG request: GET /task/active None
2023-11-04T03:59:30.289 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T03:59:31.399 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T03:59:31.400 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T03:59:31.400 ERROR:teuthology.orchestra.console.smithi099:Failed to power off
2023-11-04T03:59:31.400 INFO:teuthology.orchestra.console.smithi099:Power on
2023-11-04T03:59:31.401 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T03:59:43.135 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None
2023-11-04T03:59:43.138 DEBUG:teuthology.provision.fog.smithi081:FOG request: GET /task/active None
2023-11-04T03:59:51.470 DEBUG:teuthology.orchestra.console.smithi099:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T03:59:51.471 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T03:59:58.237 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None
2023-11-04T03:59:58.239 DEBUG:teuthology.provision.fog.smithi081:FOG request: GET /task/active None
2023-11-04T03:59:58.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:00:01.370 WARNING:teuthology.provision.fog.smithi081:[Errno None] Unable to connect to port 22 on 172.21.15.81
2023-11-04T04:00:07.374 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:00:10.426 WARNING:teuthology.provision.fog.smithi081:[Errno None] Unable to connect to port 22 on 172.21.15.81
2023-11-04T04:00:11.541 DEBUG:teuthology.orchestra.console.smithi099:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:00:11.541 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T04:00:13.317 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None
2023-11-04T04:00:13.394 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:00:16.430 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:00:16.474 WARNING:teuthology.provision.fog.smithi164:[Errno None] Unable to connect to port 22 on 172.21.15.164
2023-11-04T04:00:19.482 WARNING:teuthology.provision.fog.smithi081:[Errno None] Unable to connect to port 22 on 172.21.15.81
2023-11-04T04:00:22.478 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:00:25.486 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:00:25.530 WARNING:teuthology.provision.fog.smithi164:[Errno None] Unable to connect to port 22 on 172.21.15.164
2023-11-04T04:00:28.538 WARNING:teuthology.provision.fog.smithi081:[Errno None] Unable to connect to port 22 on 172.21.15.81
2023-11-04T04:00:31.534 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:00:31.608 DEBUG:teuthology.orchestra.console.smithi099:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:00:31.608 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:00:32.616 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:00:34.542 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:00:34.590 WARNING:teuthology.provision.fog.smithi164:[Errno None] Unable to connect to port 22 on 172.21.15.164
2023-11-04T04:00:36.618 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:00:37.594 WARNING:teuthology.provision.fog.smithi081:[Errno None] Unable to connect to port 22 on 172.21.15.81
2023-11-04T04:00:37.727 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:00:40.595 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:00:41.729 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:00:42.838 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:00:43.597 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:00:43.674 WARNING:teuthology.provision.fog.smithi164:[Errno None] Unable to connect to port 22 on 172.21.15.164
2023-11-04T04:00:46.650 WARNING:teuthology.provision.fog.smithi081:[Errno None] Unable to connect to port 22 on 172.21.15.81
2023-11-04T04:00:46.839 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:00:47.947 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:00:49.677 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:00:51.950 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:00:52.653 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:00:52.734 WARNING:teuthology.provision.fog.smithi164:[Errno None] Unable to connect to port 22 on 172.21.15.164
2023-11-04T04:00:53.059 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:00:55.706 WARNING:teuthology.provision.fog.smithi081:[Errno None] Unable to connect to port 22 on 172.21.15.81
2023-11-04T04:00:57.062 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:00:58.170 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:00:58.736 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:01:01.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:01:01.818 WARNING:teuthology.provision.fog.smithi164:[Errno None] Unable to connect to port 22 on 172.21.15.164
2023-11-04T04:01:02.172 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:01:03.281 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:01:07.282 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:01:07.820 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:01:08.391 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:01:10.878 WARNING:teuthology.provision.fog.smithi164:[Errno None] Unable to connect to port 22 on 172.21.15.164
2023-11-04T04:01:12.393 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:01:13.502 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:01:16.883 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:01:17.505 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:01:18.613 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:01:22.618 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:01:23.726 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:01:27.730 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:01:28.838 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:01:32.843 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:01:33.952 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:01:37.957 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:01:39.065 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:01:43.070 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:01:44.180 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:01:44.181 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T04:01:44.181 DEBUG:teuthology.provision.fog.smithi099:Canceling deploy task with ID 582326
2023-11-04T04:01:44.181 DEBUG:teuthology.provision.fog.smithi099:FOG request: DELETE /task/cancel {"id": 582326}
2023-11-04T04:01:44.244 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi099
2023-11-04T04:01:44.245 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T04:01:44.411 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=9fd066e7e91046f3a3bb7007d38a0be3
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi099
2023-11-04T04:01:44.492 INFO:teuthology.nuke:Checking targets against current locks
2023-11-04T04:01:44.552 DEBUG:teuthology.nuke:shortname: smithi081
2023-11-04T04:01:44.552 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-04T04:01:44.578 DEBUG:teuthology.nuke:shortname: smithi099
2023-11-04T04:01:44.579 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-04T04:01:44.580 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-04_03:36:46-smoke-reef-release-distro-default-smithi/7446770', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-04 03:57:17.152532', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD37IRqNOlXTco7tnxWLJONZTK/j85GDXdSq92Ntroba5d3a/TNLj+aXA9B4j4QuyzCuVHTo447l8oGmVmY8JaU='}
2023-11-04T04:01:44.605 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi099.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-04_03:36:46-smoke-reef-release-distro-default-smithi/7446770', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-04 03:57:17.155497', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}
2023-11-04T04:01:44.611 DEBUG:teuthology.nuke:shortname: smithi164
2023-11-04T04:01:44.611 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-04T04:01:44.640 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi164.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-04_03:36:46-smoke-reef-release-distro-default-smithi/7446770', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-04 03:57:17.154211', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDEavO115y8e47iAfWndlRb6pDswfLrmyJIaOME0n8LWJq8JkSZNDURAWR8NhDCGLNxSNqUee2OGRi5uZVkRTeQ='}
2023-11-04T04:01:44.654 INFO:teuthology.orchestra.console.smithi081:Power off
2023-11-04T04:01:44.654 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:01:44.675 DEBUG:teuthology.orchestra.console.smithi081:power off output: Chassis Power Control: Down/Off
2023-11-04T04:01:44.676 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:01:44.696 INFO:teuthology.orchestra.console.smithi099:Power off
2023-11-04T04:01:44.697 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:01:44.705 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on
2023-11-04T04:01:44.730 INFO:teuthology.orchestra.console.smithi164:Power off
2023-11-04T04:01:44.731 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:01:44.752 DEBUG:teuthology.orchestra.console.smithi164:power off output: Chassis Power Control: Down/Off
2023-11-04T04:01:44.753 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:01:44.775 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is on
2023-11-04T04:01:48.708 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:01:48.777 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:01:48.817 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on
2023-11-04T04:01:48.887 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is on
2023-11-04T04:01:52.818 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:01:52.888 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:01:52.928 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on
2023-11-04T04:01:52.997 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is on
2023-11-04T04:01:56.930 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:01:56.998 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:01:57.040 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is off
2023-11-04T04:01:57.108 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is off
2023-11-04T04:01:57.142 INFO:teuthology.orchestra.console.smithi081:Power off completed
2023-11-04T04:01:57.209 INFO:teuthology.orchestra.console.smithi164:Power off completed
2023-11-04T04:01:57.296 INFO:teuthology.lock.ops:unlocked: smithi081.front.sepia.ceph.com
2023-11-04T04:01:57.363 INFO:teuthology.lock.ops:unlocked: smithi164.front.sepia.ceph.com
2023-11-04T04:02:01.718 WARNING:teuthology.provision.fog.smithi081:timed out
2023-11-04T04:02:04.764 DEBUG:teuthology.orchestra.console.smithi099:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:02:04.764 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:02:07.724 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:02:16.894 WARNING:teuthology.provision.fog.smithi164:timed out
2023-11-04T04:02:22.898 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:02:24.832 DEBUG:teuthology.orchestra.console.smithi099:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:02:24.832 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:02:25.978 WARNING:teuthology.provision.fog.smithi164:[Errno None] Unable to connect to port 22 on 172.21.15.164
2023-11-04T04:02:26.046 WARNING:teuthology.provision.fog.smithi081:[Errno None] Unable to connect to port 22 on 172.21.15.81
2023-11-04T04:02:31.983 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:02:32.048 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:02:35.038 WARNING:teuthology.provision.fog.smithi164:[Errno None] Unable to connect to port 22 on 172.21.15.164
2023-11-04T04:02:35.098 WARNING:teuthology.provision.fog.smithi081:[Errno None] Unable to connect to port 22 on 172.21.15.81
2023-11-04T04:02:41.043 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:02:41.100 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:02:44.122 WARNING:teuthology.provision.fog.smithi164:[Errno None] Unable to connect to port 22 on 172.21.15.164
2023-11-04T04:02:44.158 WARNING:teuthology.provision.fog.smithi081:[Errno None] Unable to connect to port 22 on 172.21.15.81
2023-11-04T04:02:44.904 DEBUG:teuthology.orchestra.console.smithi099:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:02:44.905 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:02:45.916 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:02:49.918 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:02:50.125 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:02:50.160 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:02:51.027 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:02:53.178 WARNING:teuthology.provision.fog.smithi164:[Errno None] Unable to connect to port 22 on 172.21.15.164
2023-11-04T04:02:53.242 WARNING:teuthology.provision.fog.smithi081:[Errno None] Unable to connect to port 22 on 172.21.15.81
2023-11-04T04:02:55.030 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:02:56.139 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:02:59.183 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:02:59.244 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:03:00.141 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:03:01.250 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:03:02.239 WARNING:teuthology.provision.fog.smithi164:[Errno None] Unable to connect to port 22 on 172.21.15.164
2023-11-04T04:03:02.298 WARNING:teuthology.provision.fog.smithi081:[Errno None] Unable to connect to port 22 on 172.21.15.81
2023-11-04T04:03:05.254 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:03:06.362 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:03:08.243 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:03:08.300 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:03:10.365 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:03:11.322 WARNING:teuthology.provision.fog.smithi164:[Errno None] Unable to connect to port 22 on 172.21.15.164
2023-11-04T04:03:11.354 WARNING:teuthology.provision.fog.smithi081:[Errno None] Unable to connect to port 22 on 172.21.15.81
2023-11-04T04:03:11.475 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:03:15.478 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:03:16.586 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:03:17.325 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:03:17.355 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:03:20.383 WARNING:teuthology.provision.fog.smithi164:[Errno None] Unable to connect to port 22 on 172.21.15.164
2023-11-04T04:03:20.588 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:03:21.698 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:03:25.702 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:03:26.386 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:03:26.810 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:03:30.814 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:03:31.923 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:03:35.674 WARNING:teuthology.provision.fog.smithi081:[Errno None] Unable to connect to port 22 on 172.21.15.81
2023-11-04T04:03:35.925 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:03:37.034 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:03:41.038 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:03:41.678 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:03:42.148 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:03:44.730 WARNING:teuthology.provision.fog.smithi081:[Errno None] Unable to connect to port 22 on 172.21.15.81
2023-11-04T04:03:44.894 WARNING:teuthology.provision.fog.smithi164:[Errno None] Unable to connect to port 22 on 172.21.15.164
2023-11-04T04:03:46.150 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:03:47.258 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:03:50.735 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:03:50.819 ERROR:teuthology.orchestra.connection:Error authenticating with smithi081.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2023-11-04T04:03:50.896 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:03:51.259 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:03:52.368 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:03:53.978 WARNING:teuthology.provision.fog.smithi164:[Errno None] Unable to connect to port 22 on 172.21.15.164
2023-11-04T04:03:54.944 ERROR:teuthology.orchestra.connection:Error authenticating with smithi081.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2023-11-04T04:03:56.370 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:03:57.480 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-11-04T04:03:57.481 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T04:03:57.481 ERROR:teuthology.orchestra.console.smithi099:Failed to power off
2023-11-04T04:03:57.630 INFO:teuthology.lock.ops:unlocked: smithi099.front.sepia.ceph.com
2023-11-04T04:03:58.001 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi099