2023-09-21T13:07:41.625 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-21T13:07:41.627 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-21T13:07:41.822 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fda43c3d670>, signals=[15]) 2023-09-21T13:07:41.823 INFO:teuthology.lock.ops:Start node 'smithi097.front.sepia.ceph.com' reimaging 2023-09-21T13:07:41.823 INFO:teuthology.lock.ops:Updating [smithi097.front.sepia.ceph.com]: reset os type and version on server 2023-09-21T13:07:41.824 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2023-09-21T13:07:41.843 INFO:teuthology.lock.ops:Start node 'smithi103.front.sepia.ceph.com' reimaging 2023-09-21T13:07:41.843 INFO:teuthology.lock.ops:Updating [smithi103.front.sepia.ceph.com]: reset os type and version on server 2023-09-21T13:07:41.844 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2023-09-21T13:07:41.845 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /host {"name": "smithi097"} 2023-09-21T13:07:41.868 INFO:teuthology.lock.ops:Start node 'smithi104.front.sepia.ceph.com' reimaging 2023-09-21T13:07:41.868 INFO:teuthology.lock.ops:Updating [smithi104.front.sepia.ceph.com]: reset os type and version on server 2023-09-21T13:07:41.868 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2023-09-21T13:07:41.870 DEBUG:teuthology.provision.fog.smithi103:FOG request: GET /host {"name": "smithi103"} 2023-09-21T13:07:41.892 INFO:teuthology.lock.ops:Start node 'smithi125.front.sepia.ceph.com' reimaging 2023-09-21T13:07:41.893 INFO:teuthology.lock.ops:Updating [smithi125.front.sepia.ceph.com]: reset os type and version on server 2023-09-21T13:07:41.893 INFO:teuthology.lock.ops:Updating smithi125.front.sepia.ceph.com on lock server 2023-09-21T13:07:41.894 DEBUG:teuthology.provision.fog.smithi104:FOG request: GET /host {"name": "smithi104"} 2023-09-21T13:07:41.921 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /host {"name": "smithi125"} 2023-09-21T13:07:41.925 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /image {"name": "smithi_ubuntu_22.04"} 2023-09-21T13:07:41.944 DEBUG:teuthology.provision.fog.smithi103:FOG request: GET /image {"name": "smithi_ubuntu_22.04"} 2023-09-21T13:07:41.962 DEBUG:teuthology.provision.fog.smithi097:Requesting image smithi_ubuntu_22.04 (ID 6) 2023-09-21T13:07:41.962 DEBUG:teuthology.provision.fog.smithi097:FOG request: PUT /host/202 {"imageID": 6} 2023-09-21T13:07:41.969 DEBUG:teuthology.provision.fog.smithi104:FOG request: GET /image {"name": "smithi_ubuntu_22.04"} 2023-09-21T13:07:41.986 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /image {"name": "smithi_ubuntu_22.04"} 2023-09-21T13:07:41.993 DEBUG:teuthology.provision.fog.smithi103:Requesting image smithi_ubuntu_22.04 (ID 6) 2023-09-21T13:07:41.993 DEBUG:teuthology.provision.fog.smithi103:FOG request: PUT /host/208 {"imageID": 6} 2023-09-21T13:07:42.011 DEBUG:teuthology.provision.fog.smithi104:Requesting image smithi_ubuntu_22.04 (ID 6) 2023-09-21T13:07:42.011 DEBUG:teuthology.provision.fog.smithi104:FOG request: PUT /host/209 {"imageID": 6} 2023-09-21T13:07:42.018 INFO:teuthology.provision.fog.smithi097:Scheduling deploy of ubuntu 22.04 2023-09-21T13:07:42.018 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-09-21T13:07:42.029 DEBUG:teuthology.provision.fog.smithi125:Requesting image smithi_ubuntu_22.04 (ID 6) 2023-09-21T13:07:42.029 DEBUG:teuthology.provision.fog.smithi125:FOG request: PUT /host/230 {"imageID": 6} 2023-09-21T13:07:42.038 INFO:teuthology.provision.fog.smithi103:Scheduling deploy of ubuntu 22.04 2023-09-21T13:07:42.039 DEBUG:teuthology.provision.fog.smithi103:FOG request: GET /task/active None 2023-09-21T13:07:42.074 INFO:teuthology.provision.fog.smithi104:Scheduling deploy of ubuntu 22.04 2023-09-21T13:07:42.074 DEBUG:teuthology.provision.fog.smithi104:FOG request: GET /task/active None 2023-09-21T13:07:42.077 INFO:teuthology.provision.fog.smithi125:Scheduling deploy of ubuntu 22.04 2023-09-21T13:07:42.077 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /task/active None 2023-09-21T13:07:42.108 DEBUG:teuthology.provision.fog.smithi103:FOG request: GET /tasktype {"name": "deploy"} 2023-09-21T13:07:42.112 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /tasktype {"name": "deploy"} 2023-09-21T13:07:42.144 DEBUG:teuthology.provision.fog.smithi104:FOG request: GET /tasktype {"name": "deploy"} 2023-09-21T13:07:42.147 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /tasktype {"name": "deploy"} 2023-09-21T13:07:42.151 DEBUG:teuthology.provision.fog.smithi097:FOG request: POST /host/202/task {"taskTypeID": 1} 2023-09-21T13:07:42.152 DEBUG:teuthology.provision.fog.smithi103:FOG request: POST /host/208/task {"taskTypeID": 1} 2023-09-21T13:07:42.181 DEBUG:teuthology.provision.fog.smithi104:FOG request: POST /host/209/task {"taskTypeID": 1} 2023-09-21T13:07:42.202 DEBUG:teuthology.provision.fog.smithi125:FOG request: POST /host/230/task {"taskTypeID": 1} 2023-09-21T13:07:42.214 DEBUG:teuthology.provision.fog.smithi103:FOG request: GET /task/active None 2023-09-21T13:07:42.217 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-09-21T13:07:42.240 DEBUG:teuthology.provision.fog.smithi104:FOG request: GET /task/active None 2023-09-21T13:07:42.254 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /task/active None 2023-09-21T13:07:42.324 INFO:teuthology.orchestra.console.smithi103:Power off 2023-09-21T13:07:42.325 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T13:07:42.342 INFO:teuthology.orchestra.console.smithi097:Power off 2023-09-21T13:07:42.342 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T13:07:42.377 INFO:teuthology.orchestra.console.smithi104:Power off 2023-09-21T13:07:42.378 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T13:07:42.392 INFO:teuthology.orchestra.console.smithi125:Power off 2023-09-21T13:07:42.393 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T13:08:02.393 DEBUG:teuthology.orchestra.console.smithi103:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:08:02.394 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T13:08:02.408 DEBUG:teuthology.orchestra.console.smithi097:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:08:02.408 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T13:08:02.448 DEBUG:teuthology.orchestra.console.smithi104:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:08:02.448 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T13:08:02.459 DEBUG:teuthology.orchestra.console.smithi125:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:08:02.460 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T13:08:22.468 DEBUG:teuthology.orchestra.console.smithi103:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:08:22.469 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T13:08:22.478 DEBUG:teuthology.orchestra.console.smithi097:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:08:22.479 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T13:08:22.517 DEBUG:teuthology.orchestra.console.smithi104:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:08:22.518 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T13:08:22.531 DEBUG:teuthology.orchestra.console.smithi125:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:08:22.532 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T13:08:42.538 DEBUG:teuthology.orchestra.console.smithi103:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:08:42.540 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:08:42.549 DEBUG:teuthology.orchestra.console.smithi097:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:08:42.550 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:08:42.580 DEBUG:teuthology.orchestra.console.smithi104:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:08:42.581 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:08:42.600 DEBUG:teuthology.orchestra.console.smithi125:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:08:42.601 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:08:43.549 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:08:43.558 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:08:43.588 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:08:43.608 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:08:47.554 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:08:47.561 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:08:47.589 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:08:47.609 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:08:48.664 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:08:48.669 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:08:48.697 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:08:48.719 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:08:52.666 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:08:52.673 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:08:52.698 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:08:52.721 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:08:53.776 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:08:53.782 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:08:53.806 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:08:53.832 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:08:57.782 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:08:57.789 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:08:57.808 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:08:57.834 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:08:58.891 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:08:58.897 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:08:58.918 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:08:58.943 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:09:02.895 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:02.903 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:02.919 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:02.945 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:04.006 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:09:04.012 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:09:04.028 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:09:04.053 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:09:08.010 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:08.018 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:08.029 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:08.054 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:09.121 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:09:09.127 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:09:09.140 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:09:09.163 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:09:13.126 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:13.133 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:13.141 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:13.165 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:14.235 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:09:14.240 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:09:14.248 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:09:14.274 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:09:18.238 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:18.245 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:18.253 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:18.275 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:19.347 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:09:19.354 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:09:19.361 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:09:19.384 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:09:23.353 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:23.361 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:23.368 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:23.385 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:24.463 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:09:24.469 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:09:24.476 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:09:24.493 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:09:28.466 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:28.473 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:28.480 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:28.495 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:29.576 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:09:29.582 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:09:29.589 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:09:29.603 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:09:33.578 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:33.586 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:33.593 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:33.604 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:34.690 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:09:34.695 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:09:34.700 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:09:34.712 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:09:38.696 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:38.707 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:38.713 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:38.719 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:39.810 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:09:39.815 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:09:39.820 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:09:39.827 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:09:43.814 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:43.824 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:43.832 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:43.839 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:44.926 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:09:44.933 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:09:44.940 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:09:44.949 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:09:48.930 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:48.937 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:48.944 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:48.951 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:50.041 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:09:50.045 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:09:50.052 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:09:50.058 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:09:54.046 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:54.053 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:54.058 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:54.065 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:09:55.155 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:09:55.156 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds 2023-09-21T13:09:55.157 ERROR:teuthology.orchestra.console.smithi103:Failed to power off 2023-09-21T13:09:55.157 INFO:teuthology.orchestra.console.smithi103:Power on 2023-09-21T13:09:55.158 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-09-21T13:09:55.166 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:09:55.166 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds 2023-09-21T13:09:55.167 ERROR:teuthology.orchestra.console.smithi097:Failed to power off 2023-09-21T13:09:55.167 INFO:teuthology.orchestra.console.smithi097:Power on 2023-09-21T13:09:55.167 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-09-21T13:09:55.175 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:09:55.175 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds 2023-09-21T13:09:55.175 ERROR:teuthology.orchestra.console.smithi104:Failed to power off 2023-09-21T13:09:55.176 INFO:teuthology.orchestra.console.smithi104:Power on 2023-09-21T13:09:55.176 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-09-21T13:09:55.182 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:09:55.183 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds 2023-09-21T13:09:55.183 ERROR:teuthology.orchestra.console.smithi125:Failed to power off 2023-09-21T13:09:55.183 INFO:teuthology.orchestra.console.smithi125:Power on 2023-09-21T13:09:55.184 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-09-21T13:10:15.231 DEBUG:teuthology.orchestra.console.smithi103:power on output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:10:15.232 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-09-21T13:10:15.240 DEBUG:teuthology.orchestra.console.smithi097:power on output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:10:15.241 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-09-21T13:10:15.249 DEBUG:teuthology.orchestra.console.smithi104:power on output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:10:15.249 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-09-21T13:10:15.258 DEBUG:teuthology.orchestra.console.smithi125:power on output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:10:15.259 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-09-21T13:10:35.306 DEBUG:teuthology.orchestra.console.smithi103:power on output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:10:35.307 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-09-21T13:10:35.315 DEBUG:teuthology.orchestra.console.smithi097:power on output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:10:35.316 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-09-21T13:10:35.325 DEBUG:teuthology.orchestra.console.smithi104:power on output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:10:35.326 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-09-21T13:10:35.332 DEBUG:teuthology.orchestra.console.smithi125:power on output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:10:35.333 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-09-21T13:10:55.377 DEBUG:teuthology.orchestra.console.smithi103:power on output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:10:55.378 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:10:55.386 DEBUG:teuthology.orchestra.console.smithi097:power on output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:10:55.387 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:10:55.394 DEBUG:teuthology.orchestra.console.smithi104:power on output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:10:55.395 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:10:55.403 DEBUG:teuthology.orchestra.console.smithi125:power on output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:10:55.404 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:10:56.386 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:10:56.394 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:10:56.403 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:10:56.411 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:11:00.390 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:00.399 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:00.406 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:00.413 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:01.503 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:11:01.508 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:11:01.514 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:11:01.520 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:11:05.508 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:05.516 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:05.523 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:05.529 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:06.618 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:11:06.624 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:11:06.631 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:11:06.636 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:11:10.622 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:10.629 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:10.637 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:10.644 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:11.732 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:11:11.739 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:11:11.745 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:11:11.752 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:11:15.735 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:15.742 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:15.749 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:15.756 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:16.845 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:11:16.850 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:11:16.857 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:11:16.863 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:11:20.851 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:20.858 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:20.866 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:20.872 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:21.961 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:11:21.966 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:11:21.973 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:11:21.980 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:11:25.966 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:25.973 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:25.979 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:25.985 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:27.076 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:11:27.080 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:11:27.086 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:11:27.093 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:11:31.078 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:31.086 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:31.093 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:31.100 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:32.190 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:11:32.194 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:11:32.200 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:11:32.208 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:11:36.194 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:36.200 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:36.207 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:36.214 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:37.303 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:11:37.308 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:11:37.316 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:11:37.321 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:11:41.308 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:41.315 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:41.321 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:41.328 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:42.418 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:11:42.422 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:11:42.429 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:11:42.434 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:11:46.422 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:46.428 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:46.435 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:46.441 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:47.531 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:11:47.537 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:11:47.542 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:11:47.548 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:11:51.534 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:51.543 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:51.550 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:51.557 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:52.646 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:11:52.652 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:11:52.658 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:11:52.665 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:11:56.650 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:56.657 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:56.665 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:56.671 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:11:57.759 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:11:57.766 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:11:57.774 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:11:57.780 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:12:01.763 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:12:01.772 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:12:01.779 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:12:01.786 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:12:02.874 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:12:02.881 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:12:02.887 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:12:02.896 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:12:06.880 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:12:06.888 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:12:06.894 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:12:06.900 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:12:07.991 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:12:07.992 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds 2023-09-21T13:12:07.992 DEBUG:teuthology.provision.fog.smithi103:Canceling deploy task with ID 499684 2023-09-21T13:12:07.992 DEBUG:teuthology.provision.fog.smithi103:FOG request: DELETE /task/cancel {"id": 499684} 2023-09-21T13:12:07.998 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:12:07.999 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds 2023-09-21T13:12:07.999 DEBUG:teuthology.provision.fog.smithi097:Canceling deploy task with ID 499683 2023-09-21T13:12:07.999 DEBUG:teuthology.provision.fog.smithi097:FOG request: DELETE /task/cancel {"id": 499683} 2023-09-21T13:12:08.002 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:12:08.003 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds 2023-09-21T13:12:08.003 DEBUG:teuthology.provision.fog.smithi104:Canceling deploy task with ID 499685 2023-09-21T13:12:08.003 DEBUG:teuthology.provision.fog.smithi104:FOG request: DELETE /task/cancel {"id": 499685} 2023-09-21T13:12:08.008 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:12:08.009 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds 2023-09-21T13:12:08.009 DEBUG:teuthology.provision.fog.smithi125:Canceling deploy task with ID 499686 2023-09-21T13:12:08.009 DEBUG:teuthology.provision.fog.smithi125:FOG request: DELETE /task/cancel {"id": 499686} 2023-09-21T13:12:08.044 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi103 2023-09-21T13:12:08.045 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-21T13:12:08.192 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=a30c7129ee9f462b85b37ca5ad548b96 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi103 2023-09-21T13:12:08.195 INFO:teuthology.nuke:Checking targets against current locks 2023-09-21T13:12:08.262 DEBUG:teuthology.nuke:shortname: smithi097 2023-09-21T13:12:08.263 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-21T13:12:08.287 DEBUG:teuthology.nuke:shortname: smithi103 2023-09-21T13:12:08.287 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-21T13:12:08.290 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/svelar-2023-09-20_07:59:56-rados-wip-svelar-testing-1-2023-09-20-1125-distro-default-smithi/7398629', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 13:07:40.800970', 'locked_by': 'scheduled_svelar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG5PsAo33mTwEeo1Eu0EdUZ9KfA//4SDOy28XEJkXGfqNMeL3b7Qq8CbTO4CRaDEUE+6minPuzhmX0SCWe1ivyc='} 2023-09-21T13:12:08.305 DEBUG:teuthology.nuke:shortname: smithi104 2023-09-21T13:12:08.305 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-21T13:12:08.328 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/svelar-2023-09-20_07:59:56-rados-wip-svelar-testing-1-2023-09-20-1125-distro-default-smithi/7398629', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 13:07:40.797343', 'locked_by': 'scheduled_svelar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-09-21T13:12:08.338 INFO:teuthology.orchestra.console.smithi097:Power off 2023-09-21T13:12:08.339 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T13:12:08.346 DEBUG:teuthology.nuke:shortname: smithi125 2023-09-21T13:12:08.347 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-21T13:12:08.352 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/svelar-2023-09-20_07:59:56-rados-wip-svelar-testing-1-2023-09-20-1125-distro-default-smithi/7398629', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 13:07:40.798671', 'locked_by': 'scheduled_svelar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBdtk5kpPG6eYNxsbUUhP94wDXvlRmA02PaokXMYSATdfsN+ggFZAqSpAnE008SiaWHo10lbXGPIxwUzYXOv798='} 2023-09-21T13:12:08.372 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi125.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/svelar-2023-09-20_07:59:56-rados-wip-svelar-testing-1-2023-09-20-1125-distro-default-smithi/7398629', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 13:07:40.799825', 'locked_by': 'scheduled_svelar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKrKaocHzvsx1FEGng3U74HTVChGLdJF4uALWtSJ6xbpJspvUuq2DiepEzmRIrn+LgzLzLIgo9uiaz6dsWDgbHk='} 2023-09-21T13:12:08.395 INFO:teuthology.orchestra.console.smithi104:Power off 2023-09-21T13:12:08.396 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T13:12:08.411 INFO:teuthology.orchestra.console.smithi103:Power off 2023-09-21T13:12:08.412 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T13:12:08.449 INFO:teuthology.orchestra.console.smithi125:Power off 2023-09-21T13:12:08.450 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T13:12:28.409 DEBUG:teuthology.orchestra.console.smithi097:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:12:28.410 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T13:12:28.468 DEBUG:teuthology.orchestra.console.smithi104:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:12:28.468 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T13:12:28.480 DEBUG:teuthology.orchestra.console.smithi103:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:12:28.480 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T13:12:28.521 DEBUG:teuthology.orchestra.console.smithi125:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:12:28.521 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T13:12:48.481 DEBUG:teuthology.orchestra.console.smithi097:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:12:48.482 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T13:12:48.540 DEBUG:teuthology.orchestra.console.smithi104:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:12:48.540 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T13:12:48.549 DEBUG:teuthology.orchestra.console.smithi103:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:12:48.550 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T13:12:48.588 DEBUG:teuthology.orchestra.console.smithi125:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:12:48.588 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-21T13:13:08.557 DEBUG:teuthology.orchestra.console.smithi097:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:13:08.559 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:08.611 DEBUG:teuthology.orchestra.console.smithi104:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:13:08.612 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:08.624 DEBUG:teuthology.orchestra.console.smithi103:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:13:08.625 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:08.659 DEBUG:teuthology.orchestra.console.smithi125:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-09-21T13:13:08.660 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:09.569 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:13:09.623 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:13:09.634 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:13:09.669 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:13:13.571 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:13.624 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:13.635 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:13.670 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:14.682 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:13:14.734 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:13:14.743 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:13:14.779 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:13:18.686 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:18.735 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:18.744 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:18.781 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:19.796 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:13:19.845 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:13:19.853 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:13:19.890 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:13:23.799 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:23.846 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:23.854 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:23.891 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:24.910 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:13:24.954 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:13:24.963 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:13:25.002 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:13:28.915 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:28.955 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:28.964 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:29.002 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:30.026 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:13:30.064 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:13:30.072 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:13:30.111 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:13:34.030 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:34.065 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:34.073 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:34.113 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:35.140 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:13:35.176 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:13:35.181 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:13:35.221 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:13:39.142 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:39.176 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:39.183 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:39.222 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:40.254 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:13:40.286 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:13:40.291 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:13:40.330 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:13:44.258 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:44.286 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:44.293 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:44.331 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:45.366 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:13:45.394 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:13:45.403 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:13:45.441 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:13:49.368 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:49.395 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:49.404 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:49.442 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:50.479 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:13:50.504 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:13:50.513 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:13:50.553 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:13:54.482 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:54.505 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:54.514 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:54.554 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:55.596 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:13:55.616 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:13:55.622 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:13:55.664 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:13:59.598 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:59.617 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:59.625 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:13:59.665 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:14:00.708 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:14:00.726 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:14:00.734 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:14:00.775 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:14:04.710 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:14:04.727 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:14:04.736 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:14:04.776 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:14:05.820 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:14:05.837 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:14:05.845 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:14:05.885 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:14:09.822 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:14:09.838 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:14:09.846 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:14:09.887 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:14:10.933 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:14:10.948 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:14:10.954 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:14:10.996 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:14:14.938 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:14:14.950 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:14:14.957 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:14:14.997 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:14:16.049 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:14:16.059 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:14:16.068 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:14:16.107 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:14:20.054 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:14:20.061 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:14:20.069 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:14:20.108 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-21T13:14:21.164 DEBUG:teuthology.orchestra.console.smithi097:check power output: 2023-09-21T13:14:21.165 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds 2023-09-21T13:14:21.166 ERROR:teuthology.orchestra.console.smithi097:Failed to power off 2023-09-21T13:14:21.171 DEBUG:teuthology.orchestra.console.smithi104:check power output: 2023-09-21T13:14:21.171 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds 2023-09-21T13:14:21.172 ERROR:teuthology.orchestra.console.smithi104:Failed to power off 2023-09-21T13:14:21.185 DEBUG:teuthology.orchestra.console.smithi103:check power output: 2023-09-21T13:14:21.185 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds 2023-09-21T13:14:21.185 ERROR:teuthology.orchestra.console.smithi103:Failed to power off 2023-09-21T13:14:21.216 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-09-21T13:14:21.217 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds 2023-09-21T13:14:21.217 ERROR:teuthology.orchestra.console.smithi125:Failed to power off 2023-09-21T13:14:21.316 INFO:teuthology.lock.ops:unlocked: smithi097.front.sepia.ceph.com 2023-09-21T13:14:21.323 INFO:teuthology.lock.ops:unlocked: smithi104.front.sepia.ceph.com 2023-09-21T13:14:21.331 INFO:teuthology.lock.ops:unlocked: smithi103.front.sepia.ceph.com 2023-09-21T13:14:21.365 INFO:teuthology.lock.ops:unlocked: smithi125.front.sepia.ceph.com 2023-09-21T13:14:21.810 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 58, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi103