2023-09-21T11:17:43.434 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-21T11:17:43.436 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-21T11:17:43.564 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcaa6cfcca0>, signals=[15]) 2023-09-21T11:17:43.565 INFO:teuthology.lock.ops:Start node 'smithi097.front.sepia.ceph.com' reimaging 2023-09-21T11:17:43.565 INFO:teuthology.lock.ops:Updating [smithi097.front.sepia.ceph.com]: reset os type and version on server 2023-09-21T11:17:43.565 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2023-09-21T11:17:43.591 INFO:teuthology.lock.ops:Start node 'smithi125.front.sepia.ceph.com' reimaging 2023-09-21T11:17:43.591 INFO:teuthology.lock.ops:Updating [smithi125.front.sepia.ceph.com]: reset os type and version on server 2023-09-21T11:17:43.592 INFO:teuthology.lock.ops:Updating smithi125.front.sepia.ceph.com on lock server 2023-09-21T11:17:43.593 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /host {"name": "smithi097"} 2023-09-21T11:17:43.616 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /host {"name": "smithi125"} 2023-09-21T11:17:43.674 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /image {"name": "smithi_ubuntu_22.04"} 2023-09-21T11:17:43.685 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /image {"name": "smithi_ubuntu_22.04"} 2023-09-21T11:17:43.714 DEBUG:teuthology.provision.fog.smithi097:Requesting image smithi_ubuntu_22.04 (ID 6) 2023-09-21T11:17:43.714 DEBUG:teuthology.provision.fog.smithi097:FOG request: PUT /host/202 {"imageID": 6} 2023-09-21T11:17:43.729 DEBUG:teuthology.provision.fog.smithi125:Requesting image smithi_ubuntu_22.04 (ID 6) 2023-09-21T11:17:43.729 DEBUG:teuthology.provision.fog.smithi125:FOG request: PUT /host/230 {"imageID": 6} 2023-09-21T11:17:43.760 INFO:teuthology.provision.fog.smithi097:Scheduling deploy of ubuntu 22.04 2023-09-21T11:17:43.760 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-09-21T11:17:43.775 INFO:teuthology.provision.fog.smithi125:Scheduling deploy of ubuntu 22.04 2023-09-21T11:17:43.776 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /task/active None 2023-09-21T11:17:43.827 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /tasktype {"name": "deploy"} 2023-09-21T11:17:43.834 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /tasktype {"name": "deploy"} 2023-09-21T11:17:43.864 DEBUG:teuthology.provision.fog.smithi097:FOG request: POST /host/202/task {"taskTypeID": 1} 2023-09-21T11:17:43.872 DEBUG:teuthology.provision.fog.smithi125:FOG request: POST /host/230/task {"taskTypeID": 1} 2023-09-21T11:17:43.940 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-09-21T11:17:43.946 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /task/active None 2023-09-21T11:17:44.043 INFO:teuthology.orchestra.console.smithi125:Power off 2023-09-21T11:17:44.044 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T11:17:44.056 INFO:teuthology.orchestra.console.smithi097:Power off 2023-09-21T11:17:44.056 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T11:18:04.113 DEBUG:teuthology.orchestra.console.smithi125:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T11:18:04.113 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T11:18:04.124 DEBUG:teuthology.orchestra.console.smithi097:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T11:18:04.124 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T11:18:24.181 DEBUG:teuthology.orchestra.console.smithi125:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T11:18:24.182 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T11:18:24.192 DEBUG:teuthology.orchestra.console.smithi097:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T11:18:24.192 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T11:18:44.253 DEBUG:teuthology.orchestra.console.smithi125:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T11:18:44.254 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:18:44.261 DEBUG:teuthology.orchestra.console.smithi097:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T11:18:44.261 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:18:45.262 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:18:45.267 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:18:49.268 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:18:49.275 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:18:50.377 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:18:50.382 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:18:54.382 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:18:54.389 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:18:55.491 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:18:55.497 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:18:59.494 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:18:59.501 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:19:00.604 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:19:00.609 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:19:04.606 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:19:04.616 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:19:05.718 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:19:05.724 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:19:09.722 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:19:09.729 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:19:10.832 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:19:10.838 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:19:14.834 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:19:14.841 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:19:15.944 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:19:15.949 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:19:19.949 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:19:19.956 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:19:21.059 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:19:21.064 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:19:25.065 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:19:25.072 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:19:26.175 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:19:26.180 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:19:30.181 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:19:30.188 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:19:31.290 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:19:31.297 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:19:35.294 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:19:35.301 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:19:36.403 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:19:36.408 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:19:40.406 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:19:40.413 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:19:41.515 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:19:41.520 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:19:45.518 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:19:45.524 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:19:46.627 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:19:46.632 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:19:50.630 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:19:50.636 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:19:51.739 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:19:51.744 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:19:55.745 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:19:55.752 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:19:56.855 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:19:56.856 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds 2023-09-21T11:19:56.857 ERROR:teuthology.orchestra.console.smithi125:Failed to power off 2023-09-21T11:19:56.857 INFO:teuthology.orchestra.console.smithi125:Power on 2023-09-21T11:19:56.857 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-09-21T11:19:56.865 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:19:56.866 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds 2023-09-21T11:19:56.866 ERROR:teuthology.orchestra.console.smithi097:Failed to power off 2023-09-21T11:19:56.866 INFO:teuthology.orchestra.console.smithi097:Power on 2023-09-21T11:19:56.867 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-09-21T11:20:16.929 DEBUG:teuthology.orchestra.console.smithi125:power on output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T11:20:16.930 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-09-21T11:20:16.940 DEBUG:teuthology.orchestra.console.smithi097:power on output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T11:20:16.941 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-09-21T11:20:37.001 DEBUG:teuthology.orchestra.console.smithi125:power on output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T11:20:37.001 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-09-21T11:20:37.010 DEBUG:teuthology.orchestra.console.smithi097:power on output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T11:20:37.010 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-09-21T11:20:57.068 DEBUG:teuthology.orchestra.console.smithi125:power on output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T11:20:57.069 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:20:57.079 DEBUG:teuthology.orchestra.console.smithi097:power on output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T11:20:57.080 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:20:58.078 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:20:58.087 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:21:02.082 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:21:02.088 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:21:03.191 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:21:03.196 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:21:07.194 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:21:07.201 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:21:08.304 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:21:08.308 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:21:12.306 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:21:12.313 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:21:13.415 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:21:13.423 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:21:17.418 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:21:17.427 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:21:18.530 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:21:18.539 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:21:22.534 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:21:22.540 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:21:23.642 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:21:23.649 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:21:27.646 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:21:27.653 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:21:28.755 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:21:28.761 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:21:32.758 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:21:32.773 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:21:33.875 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:21:33.881 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:21:37.878 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:21:37.886 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:21:38.989 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:21:38.994 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:21:42.994 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:21:43.002 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:21:44.104 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:21:44.111 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:21:48.106 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:21:48.120 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:21:49.222 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:21:49.229 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:21:53.228 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:21:53.235 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:21:54.336 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:21:54.343 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:21:58.338 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:21:58.348 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:21:59.450 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:21:59.456 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:22:03.454 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:22:03.461 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:22:04.564 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:22:04.569 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:22:08.570 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:22:08.577 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:22:09.680 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:22:09.681 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds 2023-09-21T11:22:09.682 DEBUG:teuthology.provision.fog.smithi125:Canceling deploy task with ID 499273 2023-09-21T11:22:09.682 DEBUG:teuthology.provision.fog.smithi125:FOG request: DELETE /task/cancel {"id": 499273} 2023-09-21T11:22:09.687 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:22:09.687 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds 2023-09-21T11:22:09.687 DEBUG:teuthology.provision.fog.smithi097:Canceling deploy task with ID 499272 2023-09-21T11:22:09.688 DEBUG:teuthology.provision.fog.smithi097:FOG request: DELETE /task/cancel {"id": 499272} 2023-09-21T11:22:09.746 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi125 2023-09-21T11:22:09.747 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-21T11:22:09.890 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e52bd17518f74b3286013113f4cc397b Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi125 2023-09-21T11:22:09.892 INFO:teuthology.nuke:Checking targets against current locks 2023-09-21T11:22:09.953 DEBUG:teuthology.nuke:shortname: smithi097 2023-09-21T11:22:09.953 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-21T11:22:09.976 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-09-20_16:32:45-fs-wip-lusov-config-format-distro-default-smithi/7399762', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 11:17:42.643275', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG5PsAo33mTwEeo1Eu0EdUZ9KfA//4SDOy28XEJkXGfqNMeL3b7Qq8CbTO4CRaDEUE+6minPuzhmX0SCWe1ivyc='} 2023-09-21T11:22:09.981 DEBUG:teuthology.nuke:shortname: smithi125 2023-09-21T11:22:09.982 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-21T11:22:10.003 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi125.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-09-20_16:32:45-fs-wip-lusov-config-format-distro-default-smithi/7399762', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 11:17:42.641397', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKrKaocHzvsx1FEGng3U74HTVChGLdJF4uALWtSJ6xbpJspvUuq2DiepEzmRIrn+LgzLzLIgo9uiaz6dsWDgbHk='} 2023-09-21T11:22:10.022 INFO:teuthology.orchestra.console.smithi097:Power off 2023-09-21T11:22:10.023 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T11:22:10.060 INFO:teuthology.orchestra.console.smithi125:Power off 2023-09-21T11:22:10.061 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T11:22:30.093 DEBUG:teuthology.orchestra.console.smithi097:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T11:22:30.094 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T11:22:30.133 DEBUG:teuthology.orchestra.console.smithi125:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T11:22:30.133 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T11:22:50.165 DEBUG:teuthology.orchestra.console.smithi097:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T11:22:50.166 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T11:22:50.204 DEBUG:teuthology.orchestra.console.smithi125:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T11:22:50.204 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T11:23:10.235 DEBUG:teuthology.orchestra.console.smithi097:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T11:23:10.237 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:23:10.275 DEBUG:teuthology.orchestra.console.smithi125:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T11:23:10.276 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:23:11.245 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:23:11.284 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:23:15.248 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:23:15.285 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:23:16.358 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:23:16.395 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:23:20.362 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:23:20.396 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:23:21.473 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:23:21.506 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:23:25.478 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:23:25.507 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:23:26.588 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:23:26.617 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:23:30.594 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:23:30.618 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:23:31.704 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:23:31.727 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:23:35.707 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:23:35.728 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:23:36.818 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:23:36.838 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:23:40.824 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:23:40.839 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:23:41.933 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:23:41.947 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:23:45.939 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:23:45.948 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:23:47.050 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:23:47.056 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:23:51.055 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:23:51.064 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:23:52.166 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:23:52.173 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:23:56.172 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:23:56.180 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:23:57.282 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:23:57.289 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:24:01.286 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:24:01.293 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:24:02.396 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:24:02.401 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:24:06.398 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:24:06.405 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:24:07.508 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:24:07.516 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:24:11.510 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:24:11.517 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:24:12.620 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:24:12.625 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:24:16.625 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:24:16.633 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:24:17.735 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:24:17.742 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:24:21.741 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:24:21.748 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T11:24:22.851 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T11:24:22.852 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds 2023-09-21T11:24:22.852 ERROR:teuthology.orchestra.console.smithi097:Failed to power off 2023-09-21T11:24:22.856 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T11:24:22.857 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds 2023-09-21T11:24:22.857 ERROR:teuthology.orchestra.console.smithi125:Failed to power off 2023-09-21T11:24:23.043 INFO:teuthology.lock.ops:unlocked: smithi125.front.sepia.ceph.com 2023-09-21T11:24:23.072 INFO:teuthology.lock.ops:unlocked: smithi097.front.sepia.ceph.com 2023-09-21T11:24:23.308 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 58, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi125