2023-09-21T12:33:12.630 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-21T12:33:12.632 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-09-21T12:33:12.838 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7faa52096d30>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7faa51e36b80>, signals=[15])
2023-09-21T12:33:12.839 INFO:teuthology.lock.ops:Start node 'smithi097.front.sepia.ceph.com' reimaging
2023-09-21T12:33:12.840 INFO:teuthology.lock.ops:Updating [smithi097.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T12:33:12.840 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server
2023-09-21T12:33:12.865 INFO:teuthology.lock.ops:Start node 'smithi103.front.sepia.ceph.com' reimaging
2023-09-21T12:33:12.865 INFO:teuthology.lock.ops:Updating [smithi103.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T12:33:12.866 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server
2023-09-21T12:33:12.867 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /host {"name": "smithi097"}
2023-09-21T12:33:12.889 INFO:teuthology.lock.ops:Start node 'smithi125.front.sepia.ceph.com' reimaging
2023-09-21T12:33:12.890 INFO:teuthology.lock.ops:Updating [smithi125.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T12:33:12.890 INFO:teuthology.lock.ops:Updating smithi125.front.sepia.ceph.com on lock server
2023-09-21T12:33:12.892 DEBUG:teuthology.provision.fog.smithi103:FOG request: GET /host {"name": "smithi103"}
2023-09-21T12:33:12.914 INFO:teuthology.lock.ops:Start node 'smithi173.front.sepia.ceph.com' reimaging
2023-09-21T12:33:12.915 INFO:teuthology.lock.ops:Updating [smithi173.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T12:33:12.915 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server
2023-09-21T12:33:12.916 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /host {"name": "smithi125"}
2023-09-21T12:33:12.939 DEBUG:teuthology.provision.fog.smithi173:FOG request: GET /host {"name": "smithi173"}
2023-09-21T12:33:12.949 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /image {"name": "smithi_ubuntu_22.04"}
2023-09-21T12:33:12.965 DEBUG:teuthology.provision.fog.smithi103:FOG request: GET /image {"name": "smithi_ubuntu_22.04"}
2023-09-21T12:33:12.979 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /image {"name": "smithi_ubuntu_22.04"}
2023-09-21T12:33:12.991 DEBUG:teuthology.provision.fog.smithi097:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-09-21T12:33:12.992 DEBUG:teuthology.provision.fog.smithi097:FOG request: PUT /host/202 {"imageID": 6}
2023-09-21T12:33:13.006 DEBUG:teuthology.provision.fog.smithi103:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-09-21T12:33:13.006 DEBUG:teuthology.provision.fog.smithi103:FOG request: PUT /host/208 {"imageID": 6}
2023-09-21T12:33:13.012 DEBUG:teuthology.provision.fog.smithi173:FOG request: GET /image {"name": "smithi_ubuntu_22.04"}
2023-09-21T12:33:13.015 DEBUG:teuthology.provision.fog.smithi125:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-09-21T12:33:13.015 DEBUG:teuthology.provision.fog.smithi125:FOG request: PUT /host/230 {"imageID": 6}
2023-09-21T12:33:13.036 INFO:teuthology.provision.fog.smithi097:Scheduling deploy of ubuntu 22.04
2023-09-21T12:33:13.036 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None
2023-09-21T12:33:13.053 DEBUG:teuthology.provision.fog.smithi173:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-09-21T12:33:13.053 DEBUG:teuthology.provision.fog.smithi173:FOG request: PUT /host/278 {"imageID": 6}
2023-09-21T12:33:13.062 INFO:teuthology.provision.fog.smithi103:Scheduling deploy of ubuntu 22.04
2023-09-21T12:33:13.062 DEBUG:teuthology.provision.fog.smithi103:FOG request: GET /task/active None
2023-09-21T12:33:13.070 INFO:teuthology.provision.fog.smithi125:Scheduling deploy of ubuntu 22.04
2023-09-21T12:33:13.070 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /task/active None
2023-09-21T12:33:13.107 INFO:teuthology.provision.fog.smithi173:Scheduling deploy of ubuntu 22.04
2023-09-21T12:33:13.107 DEBUG:teuthology.provision.fog.smithi173:FOG request: GET /task/active None
2023-09-21T12:33:13.112 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T12:33:13.147 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T12:33:13.151 DEBUG:teuthology.provision.fog.smithi103:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T12:33:13.154 DEBUG:teuthology.provision.fog.smithi097:FOG request: POST /host/202/task {"taskTypeID": 1}
2023-09-21T12:33:13.189 DEBUG:teuthology.provision.fog.smithi173:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T12:33:13.193 DEBUG:teuthology.provision.fog.smithi125:FOG request: POST /host/230/task {"taskTypeID": 1}
2023-09-21T12:33:13.199 DEBUG:teuthology.provision.fog.smithi103:FOG request: POST /host/208/task {"taskTypeID": 1}
2023-09-21T12:33:13.216 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None
2023-09-21T12:33:13.241 DEBUG:teuthology.provision.fog.smithi173:FOG request: POST /host/278/task {"taskTypeID": 1}
2023-09-21T12:33:13.248 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /task/active None
2023-09-21T12:33:13.254 DEBUG:teuthology.provision.fog.smithi103:FOG request: GET /task/active None
2023-09-21T12:33:13.305 DEBUG:teuthology.provision.fog.smithi173:FOG request: GET /task/active None
2023-09-21T12:33:13.353 INFO:teuthology.orchestra.console.smithi097:Power off
2023-09-21T12:33:13.354 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:33:13.394 INFO:teuthology.orchestra.console.smithi125:Power off
2023-09-21T12:33:13.394 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:33:13.410 INFO:teuthology.orchestra.console.smithi103:Power off
2023-09-21T12:33:13.411 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:33:13.443 INFO:teuthology.orchestra.console.smithi173:Power off
2023-09-21T12:33:13.444 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:33:13.464 DEBUG:teuthology.orchestra.console.smithi173:power off output: Chassis Power Control: Down/Off
2023-09-21T12:33:13.465 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:33:13.488 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on
2023-09-21T12:33:17.492 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:33:17.601 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on
2023-09-21T12:33:21.606 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:33:21.714 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is off
2023-09-21T12:33:21.815 INFO:teuthology.orchestra.console.smithi173:Power off completed
2023-09-21T12:33:21.916 INFO:teuthology.orchestra.console.smithi173:Power on
2023-09-21T12:33:21.916 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T12:33:21.939 DEBUG:teuthology.orchestra.console.smithi173:power on output: Chassis Power Control: Up/On
2023-09-21T12:33:21.940 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:33:21.959 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is off
2023-09-21T12:33:25.963 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:33:26.072 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is off
2023-09-21T12:33:30.076 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:33:30.185 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on
2023-09-21T12:33:30.286 INFO:teuthology.orchestra.console.smithi173:Power on completed
2023-09-21T12:33:30.386 INFO:teuthology.provision.fog.smithi173:Waiting for deploy to finish
2023-09-21T12:33:30.387 DEBUG:teuthology.provision.fog.smithi173:FOG request: GET /task/active None
2023-09-21T12:33:33.426 DEBUG:teuthology.orchestra.console.smithi097:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:33:33.427 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:33:33.464 DEBUG:teuthology.orchestra.console.smithi125:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:33:33.465 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:33:33.479 DEBUG:teuthology.orchestra.console.smithi103:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:33:33.480 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:33:45.503 DEBUG:teuthology.provision.fog.smithi173:FOG request: GET /task/active None
2023-09-21T12:33:53.497 DEBUG:teuthology.orchestra.console.smithi097:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:33:53.497 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:33:53.532 DEBUG:teuthology.orchestra.console.smithi125:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:33:53.533 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:33:53.547 DEBUG:teuthology.orchestra.console.smithi103:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:33:53.548 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:34:00.609 DEBUG:teuthology.provision.fog.smithi173:FOG request: GET /task/active None
2023-09-21T12:34:13.569 DEBUG:teuthology.orchestra.console.smithi097:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:34:13.570 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:13.600 DEBUG:teuthology.orchestra.console.smithi125:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:34:13.600 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:13.618 DEBUG:teuthology.orchestra.console.smithi103:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:34:13.618 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:14.579 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:34:14.607 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:34:14.625 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:34:15.712 DEBUG:teuthology.provision.fog.smithi173:FOG request: GET /task/active None
2023-09-21T12:34:18.582 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:18.609 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:18.626 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:19.692 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:34:19.717 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:34:19.734 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:34:23.694 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:23.718 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:23.735 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:24.804 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:34:24.826 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:34:24.842 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:34:28.809 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:28.826 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:28.843 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:29.919 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:34:29.935 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:34:29.950 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:34:30.810 DEBUG:teuthology.provision.fog.smithi173:FOG request: GET /task/active None
2023-09-21T12:34:33.923 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:33.936 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:33.951 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:35.033 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:34:35.043 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:34:35.059 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:34:39.035 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:39.044 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:39.060 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:40.145 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:34:40.151 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:34:40.168 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:34:44.150 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:44.157 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:44.169 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:45.259 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:34:45.264 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:34:45.277 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:34:45.894 DEBUG:teuthology.provision.fog.smithi173:FOG request: GET /task/active None
2023-09-21T12:34:49.264 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:49.274 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:49.281 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:50.377 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:34:50.382 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:34:50.390 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:34:54.382 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:54.388 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:54.394 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:55.491 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:34:55.495 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:34:55.503 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:34:59.494 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:59.500 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:34:59.506 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:35:00.603 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:35:00.608 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:35:00.613 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:35:00.970 DEBUG:teuthology.provision.fog.smithi173:FOG request: GET /task/active None
2023-09-21T12:35:04.606 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:35:04.612 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:35:04.618 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:35:05.715 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:35:05.719 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:35:05.725 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:35:09.720 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:35:09.726 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:35:09.733 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:35:10.829 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:35:10.834 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:35:10.841 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:35:14.834 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:35:14.841 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:35:14.847 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:35:15.944 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:35:15.948 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:35:15.957 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:35:16.057 DEBUG:teuthology.provision.fog.smithi173:FOG request: GET /task/active None
2023-09-21T12:35:19.946 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:35:19.953 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:35:19.960 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:35:21.056 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:35:21.061 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:35:21.066 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:35:25.058 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:35:25.064 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:35:25.071 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:35:26.167 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:35:26.168 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T12:35:26.168 ERROR:teuthology.orchestra.console.smithi097:Failed to power off
2023-09-21T12:35:26.169 INFO:teuthology.orchestra.console.smithi097:Power on
2023-09-21T12:35:26.169 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T12:35:26.177 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:35:26.178 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T12:35:26.178 ERROR:teuthology.orchestra.console.smithi125:Failed to power off
2023-09-21T12:35:26.178 INFO:teuthology.orchestra.console.smithi125:Power on
2023-09-21T12:35:26.179 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T12:35:26.185 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:35:26.186 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T12:35:26.186 ERROR:teuthology.orchestra.console.smithi103:Failed to power off
2023-09-21T12:35:26.187 INFO:teuthology.orchestra.console.smithi103:Power on
2023-09-21T12:35:26.187 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T12:35:31.144 DEBUG:teuthology.provision.fog.smithi173:FOG request: GET /task/active None
2023-09-21T12:35:46.227 DEBUG:teuthology.provision.fog.smithi173:FOG request: GET /task/active None
2023-09-21T12:35:46.238 DEBUG:teuthology.orchestra.console.smithi097:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:35:46.238 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T12:35:46.248 DEBUG:teuthology.orchestra.console.smithi125:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:35:46.248 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T12:35:46.256 DEBUG:teuthology.orchestra.console.smithi103:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:35:46.256 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T12:36:01.316 DEBUG:teuthology.provision.fog.smithi173:FOG request: GET /task/active None
2023-09-21T12:36:06.306 DEBUG:teuthology.orchestra.console.smithi097:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:36:06.307 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T12:36:06.318 DEBUG:teuthology.orchestra.console.smithi125:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:36:06.319 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T12:36:06.326 DEBUG:teuthology.orchestra.console.smithi103:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:36:06.327 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T12:36:16.398 DEBUG:teuthology.provision.fog.smithi173:FOG request: GET /task/active None
2023-09-21T12:36:26.378 DEBUG:teuthology.orchestra.console.smithi097:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:36:26.378 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:36:26.388 DEBUG:teuthology.orchestra.console.smithi125:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:36:26.388 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:36:26.399 DEBUG:teuthology.orchestra.console.smithi103:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:36:26.400 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:36:27.387 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:36:27.395 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:36:27.406 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:36:31.391 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:36:31.399 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:36:31.407 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:36:31.464 DEBUG:teuthology.provision.fog.smithi173:FOG request: GET /task/active None
2023-09-21T12:36:32.501 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:36:32.506 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:36:32.514 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:36:36.507 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:36:36.514 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:36:36.521 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:36:37.617 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:36:37.622 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:36:37.630 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:36:41.622 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:36:41.629 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:36:41.636 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:36:42.732 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:36:42.737 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:36:42.743 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:36:46.542 DEBUG:teuthology.provision.fog.smithi173:FOG request: GET /task/active None
2023-09-21T12:36:46.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:36:46.733 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:36:46.741 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:36:46.749 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:36:47.843 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:36:47.850 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:36:47.857 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:36:51.849 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:36:51.856 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:36:51.863 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:36:52.959 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:36:52.964 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:36:52.972 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:36:56.964 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:36:56.972 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:36:56.978 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:36:58.074 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:36:58.079 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:36:58.086 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:37:02.078 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:02.085 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:02.092 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:03.188 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:37:03.192 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:37:03.199 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:37:05.054 WARNING:teuthology.provision.fog.smithi173:[Errno None] Unable to connect to port 22 on 172.21.15.173
2023-09-21T12:37:07.191 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:07.198 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:07.205 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:08.301 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:37:08.306 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:37:08.312 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:37:11.058 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:37:12.304 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:12.310 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:12.317 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:13.413 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:37:13.418 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:37:13.424 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:37:14.138 WARNING:teuthology.provision.fog.smithi173:[Errno None] Unable to connect to port 22 on 172.21.15.173
2023-09-21T12:37:17.418 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:17.424 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:17.430 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:18.528 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:37:18.531 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:37:18.537 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:37:20.142 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:37:22.531 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:22.538 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:22.544 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:23.195 WARNING:teuthology.provision.fog.smithi173:[Errno None] Unable to connect to port 22 on 172.21.15.173
2023-09-21T12:37:23.640 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:37:23.646 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:37:23.650 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:37:27.646 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:27.652 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:27.659 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:28.756 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:37:28.760 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:37:28.769 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:37:29.197 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:37:32.250 WARNING:teuthology.provision.fog.smithi173:[Errno None] Unable to connect to port 22 on 172.21.15.173
2023-09-21T12:37:32.757 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:32.764 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:32.771 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:33.867 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:37:33.872 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:37:33.878 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:37:37.872 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:37.880 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:37.887 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:38.252 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:37:38.983 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:37:38.984 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T12:37:38.984 DEBUG:teuthology.provision.fog.smithi097:Canceling deploy task with ID 499554
2023-09-21T12:37:38.984 DEBUG:teuthology.provision.fog.smithi097:FOG request: DELETE /task/cancel {"id": 499554}
2023-09-21T12:37:38.988 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:37:38.988 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T12:37:38.989 DEBUG:teuthology.provision.fog.smithi125:Canceling deploy task with ID 499555
2023-09-21T12:37:38.989 DEBUG:teuthology.provision.fog.smithi125:FOG request: DELETE /task/cancel {"id": 499555}
2023-09-21T12:37:38.994 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:37:38.994 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T12:37:38.994 DEBUG:teuthology.provision.fog.smithi103:Canceling deploy task with ID 499556
2023-09-21T12:37:38.995 DEBUG:teuthology.provision.fog.smithi103:FOG request: DELETE /task/cancel {"id": 499556}
2023-09-21T12:37:39.039 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi097
2023-09-21T12:37:39.040 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-09-21T12:37:39.176 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e8acf88aee2146648c0ae7940cefc8bd
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi097
2023-09-21T12:37:39.178 INFO:teuthology.nuke:Checking targets against current locks
2023-09-21T12:37:39.243 DEBUG:teuthology.nuke:shortname: smithi097
2023-09-21T12:37:39.243 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T12:37:39.266 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-09-20_16:32:45-fs-wip-lusov-config-format-distro-default-smithi/7399881', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 12:33:11.783903', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG5PsAo33mTwEeo1Eu0EdUZ9KfA//4SDOy28XEJkXGfqNMeL3b7Qq8CbTO4CRaDEUE+6minPuzhmX0SCWe1ivyc='}
2023-09-21T12:37:39.267 DEBUG:teuthology.nuke:shortname: smithi103
2023-09-21T12:37:39.267 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T12:37:39.290 DEBUG:teuthology.nuke:shortname: smithi125
2023-09-21T12:37:39.290 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T12:37:39.292 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-09-20_16:32:45-fs-wip-lusov-config-format-distro-default-smithi/7399881', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 12:33:11.780316', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBdtk5kpPG6eYNxsbUUhP94wDXvlRmA02PaokXMYSATdfsN+ggFZAqSpAnE008SiaWHo10lbXGPIxwUzYXOv798='}
2023-09-21T12:37:39.298 DEBUG:teuthology.nuke:shortname: smithi173
2023-09-21T12:37:39.298 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T12:37:39.317 INFO:teuthology.orchestra.console.smithi097:Power off
2023-09-21T12:37:39.317 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:37:39.326 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-09-20_16:32:45-fs-wip-lusov-config-format-distro-default-smithi/7399881', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 12:33:11.785510', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA1gldQARzl6M3N4zpRNA1tPijN7ju3QutWbf1WIs+I2RjvVOZ+v6JZQYnhLGHwUMH4l30aGYxxbT3ixBunlcZM='}
2023-09-21T12:37:39.328 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi125.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-09-20_16:32:45-fs-wip-lusov-config-format-distro-default-smithi/7399881', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 12:33:11.782248', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKrKaocHzvsx1FEGng3U74HTVChGLdJF4uALWtSJ6xbpJspvUuq2DiepEzmRIrn+LgzLzLIgo9uiaz6dsWDgbHk='}
2023-09-21T12:37:39.366 INFO:teuthology.orchestra.console.smithi103:Power off
2023-09-21T12:37:39.366 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:37:39.397 INFO:teuthology.orchestra.console.smithi125:Power off
2023-09-21T12:37:39.397 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:37:39.410 INFO:teuthology.orchestra.console.smithi173:Power off
2023-09-21T12:37:39.411 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:37:39.431 DEBUG:teuthology.orchestra.console.smithi173:power off output: Chassis Power Control: Down/Off
2023-09-21T12:37:39.432 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:39.453 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on
2023-09-21T12:37:43.458 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:37:43.568 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is off
2023-09-21T12:37:43.669 INFO:teuthology.orchestra.console.smithi173:Power off completed
2023-09-21T12:37:43.822 INFO:teuthology.lock.ops:unlocked: smithi173.front.sepia.ceph.com
2023-09-21T12:37:56.762 WARNING:teuthology.provision.fog.smithi173:[Errno None] Unable to connect to port 22 on 172.21.15.173
2023-09-21T12:37:59.389 DEBUG:teuthology.orchestra.console.smithi097:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:37:59.389 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:37:59.438 DEBUG:teuthology.orchestra.console.smithi103:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:37:59.439 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:37:59.467 DEBUG:teuthology.orchestra.console.smithi125:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:37:59.468 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:38:02.766 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:38:05.818 WARNING:teuthology.provision.fog.smithi173:[Errno None] Unable to connect to port 22 on 172.21.15.173
2023-09-21T12:38:11.825 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:38:14.906 WARNING:teuthology.provision.fog.smithi173:[Errno None] Unable to connect to port 22 on 172.21.15.173
2023-09-21T12:38:19.464 DEBUG:teuthology.orchestra.console.smithi097:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:38:19.464 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:38:19.504 DEBUG:teuthology.orchestra.console.smithi103:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:38:19.504 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:38:19.538 DEBUG:teuthology.orchestra.console.smithi125:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:38:19.539 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:38:20.909 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:38:23.962 WARNING:teuthology.provision.fog.smithi173:[Errno None] Unable to connect to port 22 on 172.21.15.173
2023-09-21T12:38:29.966 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:38:33.018 WARNING:teuthology.provision.fog.smithi173:[Errno None] Unable to connect to port 22 on 172.21.15.173
2023-09-21T12:38:39.022 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:38:39.537 DEBUG:teuthology.orchestra.console.smithi097:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:38:39.537 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:38:39.571 DEBUG:teuthology.orchestra.console.smithi103:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:38:39.572 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:38:39.607 DEBUG:teuthology.orchestra.console.smithi125:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:38:39.608 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:38:40.548 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:38:40.581 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:38:40.616 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:38:42.074 WARNING:teuthology.provision.fog.smithi173:[Errno None] Unable to connect to port 22 on 172.21.15.173
2023-09-21T12:38:44.549 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:38:44.582 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:38:44.616 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:38:45.660 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:38:45.693 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:38:45.725 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:38:48.078 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:38:49.663 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:38:49.695 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:38:49.726 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:38:50.774 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:38:50.804 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:38:50.836 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:38:51.134 WARNING:teuthology.provision.fog.smithi173:[Errno None] Unable to connect to port 22 on 172.21.15.173
2023-09-21T12:38:54.778 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:38:54.806 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:38:54.837 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:38:55.888 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:38:55.914 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:38:55.947 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:38:57.138 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:38:59.890 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:38:59.916 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:38:59.948 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:00.218 WARNING:teuthology.provision.fog.smithi173:[Errno None] Unable to connect to port 22 on 172.21.15.173
2023-09-21T12:39:01.000 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:39:01.025 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:39:01.059 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:39:05.002 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:05.026 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:05.060 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:06.112 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:39:06.137 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:39:06.168 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:39:06.220 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:39:09.278 WARNING:teuthology.provision.fog.smithi173:[Errno None] Unable to connect to port 22 on 172.21.15.173
2023-09-21T12:39:10.114 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:10.138 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:10.170 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:11.225 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:39:11.249 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:39:11.280 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:39:15.230 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:15.250 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:15.280 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:39:15.283 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:16.341 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:39:16.358 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:39:16.391 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:39:18.366 WARNING:teuthology.provision.fog.smithi173:[Errno None] Unable to connect to port 22 on 172.21.15.173
2023-09-21T12:39:20.344 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:20.359 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:20.393 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:21.454 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:39:21.468 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:39:21.500 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:39:24.370 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:39:25.456 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:25.468 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:25.501 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:26.567 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:39:26.577 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:39:26.614 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:39:27.450 WARNING:teuthology.provision.fog.smithi173:[Errno None] Unable to connect to port 22 on 172.21.15.173
2023-09-21T12:39:30.572 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:30.579 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:30.615 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:31.682 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:39:31.687 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:39:31.724 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:39:33.453 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:39:35.686 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:35.693 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:35.725 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:36.506 WARNING:teuthology.provision.fog.smithi173:[Errno None] Unable to connect to port 22 on 172.21.15.173
2023-09-21T12:39:36.795 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:39:36.802 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:39:36.836 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:39:40.798 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:40.805 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:40.838 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:41.908 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:39:41.913 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:39:41.948 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:39:42.509 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:39:45.566 WARNING:teuthology.provision.fog.smithi173:[Errno None] Unable to connect to port 22 on 172.21.15.173
2023-09-21T12:39:45.910 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:45.918 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:45.950 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:47.021 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:39:47.026 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:39:47.058 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:39:51.026 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:51.034 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:51.060 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:39:51.569 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:39:52.136 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-09-21T12:39:52.136 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T12:39:52.137 ERROR:teuthology.orchestra.console.smithi097:Failed to power off
2023-09-21T12:39:52.142 DEBUG:teuthology.orchestra.console.smithi103:check power output: 
2023-09-21T12:39:52.142 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T12:39:52.143 ERROR:teuthology.orchestra.console.smithi103:Failed to power off
2023-09-21T12:39:52.169 DEBUG:teuthology.orchestra.console.smithi125:check power output: 
2023-09-21T12:39:52.169 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T12:39:52.169 ERROR:teuthology.orchestra.console.smithi125:Failed to power off
2023-09-21T12:39:52.298 INFO:teuthology.lock.ops:unlocked: smithi097.front.sepia.ceph.com
2023-09-21T12:39:52.304 INFO:teuthology.lock.ops:unlocked: smithi103.front.sepia.ceph.com
2023-09-21T12:39:52.320 INFO:teuthology.lock.ops:unlocked: smithi125.front.sepia.ceph.com
2023-09-21T12:39:52.732 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi097