2023-11-04T05:58:01.588 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2
2023-11-04T05:58:01.590 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T05:58:01.780 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7fe96b75db20>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7fe96b00dca0>, signals=[15])
2023-11-04T05:58:01.782 INFO:teuthology.lock.ops:Start node 'smithi088.front.sepia.ceph.com' reimaging
2023-11-04T05:58:01.783 INFO:teuthology.lock.ops:Updating [smithi088.front.sepia.ceph.com]: reset os type and version on server
2023-11-04T05:58:01.783 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server
2023-11-04T05:58:01.802 INFO:teuthology.lock.ops:Start node 'smithi116.front.sepia.ceph.com' reimaging
2023-11-04T05:58:01.802 INFO:teuthology.lock.ops:Updating [smithi116.front.sepia.ceph.com]: reset os type and version on server
2023-11-04T05:58:01.803 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server
2023-11-04T05:58:01.804 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /host {"name": "smithi088"}
2023-11-04T05:58:01.827 INFO:teuthology.lock.ops:Start node 'smithi175.front.sepia.ceph.com' reimaging
2023-11-04T05:58:01.827 INFO:teuthology.lock.ops:Updating [smithi175.front.sepia.ceph.com]: reset os type and version on server
2023-11-04T05:58:01.828 INFO:teuthology.lock.ops:Updating smithi175.front.sepia.ceph.com on lock server
2023-11-04T05:58:01.829 DEBUG:teuthology.provision.fog.smithi116:FOG request: GET /host {"name": "smithi116"}
2023-11-04T05:58:01.852 DEBUG:teuthology.provision.fog.smithi175:FOG request: GET /host {"name": "smithi175"}
2023-11-04T05:58:01.875 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-11-04T05:58:01.898 DEBUG:teuthology.provision.fog.smithi116:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-11-04T05:58:01.912 DEBUG:teuthology.provision.fog.smithi088:Requesting image smithi_rhel_8.6 (ID 5)
2023-11-04T05:58:01.912 DEBUG:teuthology.provision.fog.smithi088:FOG request: PUT /host/193 {"imageID": 5}
2023-11-04T05:58:01.916 DEBUG:teuthology.provision.fog.smithi175:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-11-04T05:58:01.934 DEBUG:teuthology.provision.fog.smithi116:Requesting image smithi_rhel_8.6 (ID 5)
2023-11-04T05:58:01.935 DEBUG:teuthology.provision.fog.smithi116:FOG request: PUT /host/221 {"imageID": 5}
2023-11-04T05:58:01.972 DEBUG:teuthology.provision.fog.smithi175:Requesting image smithi_rhel_8.6 (ID 5)
2023-11-04T05:58:01.972 DEBUG:teuthology.provision.fog.smithi175:FOG request: PUT /host/280 {"imageID": 5}
2023-11-04T05:58:01.977 INFO:teuthology.provision.fog.smithi088:Scheduling deploy of rhel 8.6
2023-11-04T05:58:01.978 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-04T05:58:01.982 INFO:teuthology.provision.fog.smithi116:Scheduling deploy of rhel 8.6
2023-11-04T05:58:01.983 DEBUG:teuthology.provision.fog.smithi116:FOG request: GET /task/active None
2023-11-04T05:58:02.027 INFO:teuthology.provision.fog.smithi175:Scheduling deploy of rhel 8.6
2023-11-04T05:58:02.028 DEBUG:teuthology.provision.fog.smithi175:FOG request: GET /task/active None
2023-11-04T05:58:02.051 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /tasktype {"name": "deploy"}
2023-11-04T05:58:02.067 DEBUG:teuthology.provision.fog.smithi116:FOG request: GET /tasktype {"name": "deploy"}
2023-11-04T05:58:02.088 DEBUG:teuthology.provision.fog.smithi088:FOG request: POST /host/193/task {"taskTypeID": 1}
2023-11-04T05:58:02.095 DEBUG:teuthology.provision.fog.smithi175:FOG request: GET /tasktype {"name": "deploy"}
2023-11-04T05:58:02.106 DEBUG:teuthology.provision.fog.smithi116:FOG request: POST /host/221/task {"taskTypeID": 1}
2023-11-04T05:58:02.127 DEBUG:teuthology.provision.fog.smithi175:FOG request: POST /host/280/task {"taskTypeID": 1}
2023-11-04T05:58:02.144 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-04T05:58:02.163 DEBUG:teuthology.provision.fog.smithi116:FOG request: GET /task/active None
2023-11-04T05:58:02.190 DEBUG:teuthology.provision.fog.smithi175:FOG request: GET /task/active None
2023-11-04T05:58:02.254 INFO:teuthology.orchestra.console.smithi088:Power off
2023-11-04T05:58:02.255 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T05:58:02.273 INFO:teuthology.orchestra.console.smithi116:Power off
2023-11-04T05:58:02.274 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T05:58:02.282 DEBUG:teuthology.orchestra.console.smithi088:power off output: Chassis Power Control: Down/Off
2023-11-04T05:58:02.283 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:58:02.308 INFO:teuthology.orchestra.console.smithi175:Power off
2023-11-04T05:58:02.309 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T05:58:02.318 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on
2023-11-04T05:58:02.331 DEBUG:teuthology.orchestra.console.smithi175:power off output: Chassis Power Control: Down/Off
2023-11-04T05:58:02.331 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:58:02.353 DEBUG:teuthology.orchestra.console.smithi175:check power output: Chassis Power is on
2023-11-04T05:58:06.322 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:58:06.354 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:58:06.432 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on
2023-11-04T05:58:06.463 DEBUG:teuthology.orchestra.console.smithi175:check power output: Chassis Power is on
2023-11-04T05:58:10.436 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:58:10.465 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:58:10.546 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on
2023-11-04T05:58:10.575 DEBUG:teuthology.orchestra.console.smithi175:check power output: Chassis Power is off
2023-11-04T05:58:10.676 INFO:teuthology.orchestra.console.smithi175:Power off completed
2023-11-04T05:58:10.777 INFO:teuthology.orchestra.console.smithi175:Power on
2023-11-04T05:58:10.777 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T05:58:10.799 DEBUG:teuthology.orchestra.console.smithi175:power on output: Chassis Power Control: Up/On
2023-11-04T05:58:10.800 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:58:10.821 DEBUG:teuthology.orchestra.console.smithi175:check power output: Chassis Power is off
2023-11-04T05:58:14.550 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:58:14.660 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is off
2023-11-04T05:58:14.761 INFO:teuthology.orchestra.console.smithi088:Power off completed
2023-11-04T05:58:14.822 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:58:14.862 INFO:teuthology.orchestra.console.smithi088:Power on
2023-11-04T05:58:14.862 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T05:58:14.885 DEBUG:teuthology.orchestra.console.smithi088:power on output: Chassis Power Control: Up/On
2023-11-04T05:58:14.885 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:58:14.909 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is off
2023-11-04T05:58:14.931 DEBUG:teuthology.orchestra.console.smithi175:check power output: Chassis Power is on
2023-11-04T05:58:15.032 INFO:teuthology.orchestra.console.smithi175:Power on completed
2023-11-04T05:58:15.133 INFO:teuthology.provision.fog.smithi175:Waiting for deploy to finish
2023-11-04T05:58:15.134 DEBUG:teuthology.provision.fog.smithi175:FOG request: GET /task/active None
2023-11-04T05:58:18.914 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:58:19.022 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is off
2023-11-04T05:58:22.340 DEBUG:teuthology.orchestra.console.smithi116:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:58:22.340 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T05:58:23.025 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:58:23.135 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on
2023-11-04T05:58:23.236 INFO:teuthology.orchestra.console.smithi088:Power on completed
2023-11-04T05:58:23.337 INFO:teuthology.provision.fog.smithi088:Waiting for deploy to finish
2023-11-04T05:58:23.337 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-04T05:58:30.238 DEBUG:teuthology.provision.fog.smithi175:FOG request: GET /task/active None
2023-11-04T05:58:38.422 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-04T05:58:42.408 DEBUG:teuthology.orchestra.console.smithi116:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:58:42.409 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T05:58:45.326 DEBUG:teuthology.provision.fog.smithi175:FOG request: GET /task/active None
2023-11-04T05:58:53.510 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-04T05:59:00.418 DEBUG:teuthology.provision.fog.smithi175:FOG request: GET /task/active None
2023-11-04T05:59:02.482 DEBUG:teuthology.orchestra.console.smithi116:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T05:59:02.482 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:59:03.491 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:59:07.496 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:59:08.604 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-04T05:59:08.606 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:59:12.610 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:59:13.718 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:59:15.518 DEBUG:teuthology.provision.fog.smithi175:FOG request: GET /task/active None
2023-11-04T05:59:17.722 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:59:18.831 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:59:22.834 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:59:23.678 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-04T05:59:23.942 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:59:27.946 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:59:29.054 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:59:30.610 DEBUG:teuthology.provision.fog.smithi175:FOG request: GET /task/active None
2023-11-04T05:59:33.058 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:59:34.167 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:59:38.170 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:59:38.754 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-04T05:59:39.279 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:59:43.283 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:59:44.392 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:59:45.698 DEBUG:teuthology.provision.fog.smithi175:FOG request: GET /task/active None
2023-11-04T05:59:48.395 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:59:49.504 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:59:53.506 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:59:53.824 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-04T05:59:54.615 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T05:59:58.621 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T05:59:59.731 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:00:00.779 DEBUG:teuthology.provision.fog.smithi175:FOG request: GET /task/active None
2023-11-04T06:00:03.734 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:00:04.843 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:00:08.846 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:00:08.908 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-04T06:00:09.955 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:00:13.958 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:00:15.067 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:00:15.067 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T06:00:15.068 ERROR:teuthology.orchestra.console.smithi116:Failed to power off
2023-11-04T06:00:15.068 INFO:teuthology.orchestra.console.smithi116:Power on
2023-11-04T06:00:15.068 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T06:00:15.868 DEBUG:teuthology.provision.fog.smithi175:FOG request: GET /task/active None
2023-11-04T06:00:24.006 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-04T06:00:30.962 DEBUG:teuthology.provision.fog.smithi175:FOG request: GET /task/active None
2023-11-04T06:00:35.136 DEBUG:teuthology.orchestra.console.smithi116:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T06:00:35.136 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T06:00:39.090 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-04T06:00:46.055 DEBUG:teuthology.provision.fog.smithi175:FOG request: GET /task/active None
2023-11-04T06:00:54.183 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-04T06:00:55.206 DEBUG:teuthology.orchestra.console.smithi116:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T06:00:55.207 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T06:01:01.138 DEBUG:teuthology.provision.fog.smithi175:FOG request: GET /task/active None
2023-11-04T06:01:01.214 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:01:04.282 WARNING:teuthology.provision.fog.smithi175:[Errno None] Unable to connect to port 22 on 172.21.15.175
2023-11-04T06:01:09.262 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-04T06:01:10.285 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:01:13.338 WARNING:teuthology.provision.fog.smithi175:[Errno None] Unable to connect to port 22 on 172.21.15.175
2023-11-04T06:01:15.278 DEBUG:teuthology.orchestra.console.smithi116:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T06:01:15.279 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:01:16.286 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:01:19.343 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:01:20.289 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:01:21.397 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:01:22.398 WARNING:teuthology.provision.fog.smithi175:[Errno None] Unable to connect to port 22 on 172.21.15.175
2023-11-04T06:01:24.336 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-04T06:01:24.406 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:01:25.400 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:01:26.509 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:01:27.486 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-04T06:01:28.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:01:30.513 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:01:31.486 WARNING:teuthology.provision.fog.smithi175:[Errno None] Unable to connect to port 22 on 172.21.15.175
2023-11-04T06:01:31.621 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:01:33.490 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:01:35.622 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:01:36.574 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-04T06:01:36.730 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:01:37.489 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:01:40.574 WARNING:teuthology.provision.fog.smithi175:[Errno None] Unable to connect to port 22 on 172.21.15.175
2023-11-04T06:01:40.731 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:01:41.841 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:01:42.577 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:01:45.658 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-04T06:01:45.842 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:01:46.577 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:01:46.950 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:01:49.658 WARNING:teuthology.provision.fog.smithi175:[Errno None] Unable to connect to port 22 on 172.21.15.175
2023-11-04T06:01:50.952 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:01:51.661 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:01:52.061 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:01:54.714 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-04T06:01:55.661 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:01:56.063 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:01:57.171 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:01:58.714 WARNING:teuthology.provision.fog.smithi175:[Errno None] Unable to connect to port 22 on 172.21.15.175
2023-11-04T06:02:00.718 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:02:01.173 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:02:02.282 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:02:03.775 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-04T06:02:04.717 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:02:06.284 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:02:07.393 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:02:09.779 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:02:11.396 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:02:12.503 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:02:12.862 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-04T06:02:16.531 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:02:17.639 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:02:18.864 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:02:21.643 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:02:21.918 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-04T06:02:22.753 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:02:26.755 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:02:27.864 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:02:27.865 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T06:02:27.865 DEBUG:teuthology.provision.fog.smithi116:Canceling deploy task with ID 582818
2023-11-04T06:02:27.865 DEBUG:teuthology.provision.fog.smithi116:FOG request: DELETE /task/cancel {"id": 582818}
2023-11-04T06:02:27.920 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:02:27.924 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi116
2023-11-04T06:02:27.925 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T06:02:28.084 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3378ee161dfd4714b5a960224836b2b8
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi116
2023-11-04T06:02:28.087 INFO:teuthology.nuke:Checking targets against current locks
2023-11-04T06:02:28.146 DEBUG:teuthology.nuke:shortname: smithi088
2023-11-04T06:02:28.147 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-04T06:02:28.178 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-11-04_04:30:51-fs-rishabh-2023nov3-testing-default-smithi/7447180', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-04 05:58:00.655023', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE9Gn8bAfGlt8c7Sa43eBA9zALzV/6vxwY4WcJLFAvWzMI/F8qWFBnxRfQQE2pc22c7JnDSYfHD6ZCuugJnBoak='}
2023-11-04T06:02:28.179 DEBUG:teuthology.nuke:shortname: smithi116
2023-11-04T06:02:28.180 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-04T06:02:28.202 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-11-04_04:30:51-fs-rishabh-2023nov3-testing-default-smithi/7447180', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-04 05:58:00.652191', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOQx0T9Cy3PuRDcKCphGZ9Vc5tqz8pcCkm3+TX+0iQMhDeU+6rMaTDJbsjF6dAqRzIoF4739qQAYE59ailaAJfs='}
2023-11-04T06:02:28.206 DEBUG:teuthology.nuke:shortname: smithi175
2023-11-04T06:02:28.206 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-04T06:02:28.233 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi175.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-11-04_04:30:51-fs-rishabh-2023nov3-testing-default-smithi/7447180', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-04 05:58:00.653772', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}
2023-11-04T06:02:28.239 INFO:teuthology.orchestra.console.smithi088:Power off
2023-11-04T06:02:28.240 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T06:02:28.264 DEBUG:teuthology.orchestra.console.smithi088:power off output: Chassis Power Control: Down/Off
2023-11-04T06:02:28.265 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:02:28.284 INFO:teuthology.orchestra.console.smithi116:Power off
2023-11-04T06:02:28.285 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T06:02:28.297 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on
2023-11-04T06:02:28.323 INFO:teuthology.orchestra.console.smithi175:Power off
2023-11-04T06:02:28.324 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T06:02:28.346 DEBUG:teuthology.orchestra.console.smithi175:power off output: Chassis Power Control: Down/Off
2023-11-04T06:02:28.347 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:02:28.370 DEBUG:teuthology.orchestra.console.smithi175:check power output: Chassis Power is on
2023-11-04T06:02:30.971 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-04T06:02:32.298 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:02:32.371 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:02:32.407 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is off
2023-11-04T06:02:32.480 DEBUG:teuthology.orchestra.console.smithi175:check power output: Chassis Power is on
2023-11-04T06:02:32.508 INFO:teuthology.orchestra.console.smithi088:Power off completed
2023-11-04T06:02:32.666 INFO:teuthology.lock.ops:unlocked: smithi088.front.sepia.ceph.com
2023-11-04T06:02:36.482 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:02:36.592 DEBUG:teuthology.orchestra.console.smithi175:check power output: Chassis Power is on
2023-11-04T06:02:36.974 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:02:40.026 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-04T06:02:40.594 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:02:40.704 DEBUG:teuthology.orchestra.console.smithi175:check power output: Chassis Power is off
2023-11-04T06:02:40.805 INFO:teuthology.orchestra.console.smithi175:Power off completed
2023-11-04T06:02:40.964 INFO:teuthology.lock.ops:unlocked: smithi175.front.sepia.ceph.com
2023-11-04T06:02:46.031 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:02:48.356 DEBUG:teuthology.orchestra.console.smithi116:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T06:02:48.356 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T06:02:49.082 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-04T06:02:55.090 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:02:58.170 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-04T06:03:04.176 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:03:04.720 WARNING:teuthology.provision.fog.smithi175:timed out
2023-11-04T06:03:07.258 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-04T06:03:08.429 DEBUG:teuthology.orchestra.console.smithi116:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T06:03:08.430 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T06:03:10.725 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:03:13.263 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:03:16.318 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-04T06:03:22.323 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:03:25.406 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-04T06:03:28.504 DEBUG:teuthology.orchestra.console.smithi116:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T06:03:28.505 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:03:29.274 WARNING:teuthology.provision.fog.smithi175:[Errno None] Unable to connect to port 22 on 172.21.15.175
2023-11-04T06:03:29.513 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:03:31.410 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:03:33.515 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:03:34.494 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-04T06:03:34.624 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:03:35.277 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:03:38.330 WARNING:teuthology.provision.fog.smithi175:[Errno None] Unable to connect to port 22 on 172.21.15.175
2023-11-04T06:03:38.626 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:03:39.734 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:03:40.498 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:03:43.582 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-04T06:03:43.736 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:03:44.333 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:03:44.846 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:03:47.390 WARNING:teuthology.provision.fog.smithi175:[Errno None] Unable to connect to port 22 on 172.21.15.175
2023-11-04T06:03:48.847 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:03:49.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:03:49.956 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:03:52.666 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-04T06:03:53.392 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:03:53.958 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:03:55.067 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:03:56.474 WARNING:teuthology.provision.fog.smithi175:[Errno None] Unable to connect to port 22 on 172.21.15.175
2023-11-04T06:03:58.670 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:03:59.070 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:04:00.178 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:04:01.726 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-04T06:04:02.477 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:04:04.182 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:04:05.290 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:04:05.534 WARNING:teuthology.provision.fog.smithi175:[Errno None] Unable to connect to port 22 on 172.21.15.175
2023-11-04T06:04:07.731 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:04:09.294 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:04:10.402 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:04:10.814 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-04T06:04:11.536 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:04:14.406 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:04:14.618 WARNING:teuthology.provision.fog.smithi175:[Errno None] Unable to connect to port 22 on 172.21.15.175
2023-11-04T06:04:15.517 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:04:16.818 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:04:19.520 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:04:19.898 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-04T06:04:20.622 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:04:20.629 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:04:23.674 WARNING:teuthology.provision.fog.smithi175:[Errno None] Unable to connect to port 22 on 172.21.15.175
2023-11-04T06:04:24.630 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:04:25.740 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:04:25.901 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:04:28.954 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-04T06:04:29.676 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:04:29.742 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:04:30.851 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:04:32.730 WARNING:teuthology.provision.fog.smithi175:[Errno None] Unable to connect to port 22 on 172.21.15.175
2023-11-04T06:04:34.854 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:04:34.957 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:04:35.964 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:04:38.010 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-04T06:04:38.733 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T06:04:39.966 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T06:04:41.074 DEBUG:teuthology.orchestra.console.smithi116:check power output: 
2023-11-04T06:04:41.075 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T06:04:41.075 ERROR:teuthology.orchestra.console.smithi116:Failed to power off
2023-11-04T06:04:41.239 INFO:teuthology.lock.ops:unlocked: smithi116.front.sepia.ceph.com
2023-11-04T06:04:41.469 ERROR:teuthology.dispatcher.supervisor:Reimage failed 10 times ... marking machine down
2023-11-04T06:04:41.588 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi116