2023-08-16T04:24:21.737 INFO:root:teuthology version: 0.0.1.dev183+g5435569
2023-08-16T04:24:21.739 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-08-16T04:24:23.188 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7fb9646df7f0>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7fb965ba0280>, signals=[15])
2023-08-16T04:24:23.190 INFO:teuthology.lock.ops:Start node 'smithi027.front.sepia.ceph.com' reimaging
2023-08-16T04:24:23.190 INFO:teuthology.lock.ops:Updating [smithi027.front.sepia.ceph.com]: reset os type and version on server
2023-08-16T04:24:23.191 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server
2023-08-16T04:24:23.211 INFO:teuthology.lock.ops:Start node 'smithi061.front.sepia.ceph.com' reimaging
2023-08-16T04:24:23.212 INFO:teuthology.lock.ops:Updating [smithi061.front.sepia.ceph.com]: reset os type and version on server
2023-08-16T04:24:23.212 INFO:teuthology.lock.ops:Updating smithi061.front.sepia.ceph.com on lock server
2023-08-16T04:24:23.215 DEBUG:teuthology.provision.fog.smithi027:FOG request: GET /host {"name": "smithi027"}
2023-08-16T04:24:23.235 INFO:teuthology.lock.ops:Start node 'smithi105.front.sepia.ceph.com' reimaging
2023-08-16T04:24:23.236 INFO:teuthology.lock.ops:Updating [smithi105.front.sepia.ceph.com]: reset os type and version on server
2023-08-16T04:24:23.236 INFO:teuthology.lock.ops:Updating smithi105.front.sepia.ceph.com on lock server
2023-08-16T04:24:23.238 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /host {"name": "smithi061"}
2023-08-16T04:24:23.260 INFO:teuthology.lock.ops:Start node 'smithi142.front.sepia.ceph.com' reimaging
2023-08-16T04:24:23.260 INFO:teuthology.lock.ops:Updating [smithi142.front.sepia.ceph.com]: reset os type and version on server
2023-08-16T04:24:23.261 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server
2023-08-16T04:24:23.262 DEBUG:teuthology.provision.fog.smithi105:FOG request: GET /host {"name": "smithi105"}
2023-08-16T04:24:23.285 INFO:teuthology.lock.ops:Start node 'smithi163.front.sepia.ceph.com' reimaging
2023-08-16T04:24:23.285 INFO:teuthology.lock.ops:Updating [smithi163.front.sepia.ceph.com]: reset os type and version on server
2023-08-16T04:24:23.286 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server
2023-08-16T04:24:23.287 DEBUG:teuthology.provision.fog.smithi142:FOG request: GET /host {"name": "smithi142"}
2023-08-16T04:24:23.302 DEBUG:teuthology.provision.fog.smithi027:FOG request: GET /image {"name": "smithi_ubuntu_18.04"}
2023-08-16T04:24:23.310 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /host {"name": "smithi163"}
2023-08-16T04:24:23.313 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /image {"name": "smithi_ubuntu_18.04"}
2023-08-16T04:24:23.327 DEBUG:teuthology.provision.fog.smithi105:FOG request: GET /image {"name": "smithi_ubuntu_18.04"}
2023-08-16T04:24:23.352 DEBUG:teuthology.provision.fog.smithi061:Requesting image smithi_ubuntu_18.04 (ID 1)
2023-08-16T04:24:23.352 DEBUG:teuthology.provision.fog.smithi061:FOG request: PUT /host/166 {"imageID": 1}
2023-08-16T04:24:23.356 DEBUG:teuthology.provision.fog.smithi027:Requesting image smithi_ubuntu_18.04 (ID 1)
2023-08-16T04:24:23.356 DEBUG:teuthology.provision.fog.smithi027:FOG request: PUT /host/132 {"imageID": 1}
2023-08-16T04:24:23.361 DEBUG:teuthology.provision.fog.smithi142:FOG request: GET /image {"name": "smithi_ubuntu_18.04"}
2023-08-16T04:24:23.370 DEBUG:teuthology.provision.fog.smithi105:Requesting image smithi_ubuntu_18.04 (ID 1)
2023-08-16T04:24:23.370 DEBUG:teuthology.provision.fog.smithi105:FOG request: PUT /host/210 {"imageID": 1}
2023-08-16T04:24:23.388 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /image {"name": "smithi_ubuntu_18.04"}
2023-08-16T04:24:23.391 INFO:teuthology.provision.fog.smithi061:Scheduling deploy of ubuntu 18.04
2023-08-16T04:24:23.392 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None
2023-08-16T04:24:23.407 DEBUG:teuthology.provision.fog.smithi142:Requesting image smithi_ubuntu_18.04 (ID 1)
2023-08-16T04:24:23.408 DEBUG:teuthology.provision.fog.smithi142:FOG request: PUT /host/247 {"imageID": 1}
2023-08-16T04:24:23.414 INFO:teuthology.provision.fog.smithi027:Scheduling deploy of ubuntu 18.04
2023-08-16T04:24:23.414 DEBUG:teuthology.provision.fog.smithi027:FOG request: GET /task/active None
2023-08-16T04:24:23.416 INFO:teuthology.provision.fog.smithi105:Scheduling deploy of ubuntu 18.04
2023-08-16T04:24:23.417 DEBUG:teuthology.provision.fog.smithi105:FOG request: GET /task/active None
2023-08-16T04:24:23.430 DEBUG:teuthology.provision.fog.smithi163:Requesting image smithi_ubuntu_18.04 (ID 1)
2023-08-16T04:24:23.431 DEBUG:teuthology.provision.fog.smithi163:FOG request: PUT /host/268 {"imageID": 1}
2023-08-16T04:24:23.460 INFO:teuthology.provision.fog.smithi142:Scheduling deploy of ubuntu 18.04
2023-08-16T04:24:23.461 DEBUG:teuthology.provision.fog.smithi142:FOG request: GET /task/active None
2023-08-16T04:24:23.464 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /tasktype {"name": "deploy"}
2023-08-16T04:24:23.477 DEBUG:teuthology.provision.fog.smithi105:FOG request: GET /tasktype {"name": "deploy"}
2023-08-16T04:24:23.480 DEBUG:teuthology.provision.fog.smithi027:FOG request: GET /tasktype {"name": "deploy"}
2023-08-16T04:24:23.482 INFO:teuthology.provision.fog.smithi163:Scheduling deploy of ubuntu 18.04
2023-08-16T04:24:23.483 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /task/active None
2023-08-16T04:24:23.505 DEBUG:teuthology.provision.fog.smithi061:FOG request: POST /host/166/task {"taskTypeID": 1}
2023-08-16T04:24:23.517 DEBUG:teuthology.provision.fog.smithi142:FOG request: GET /tasktype {"name": "deploy"}
2023-08-16T04:24:23.520 DEBUG:teuthology.provision.fog.smithi105:FOG request: POST /host/210/task {"taskTypeID": 1}
2023-08-16T04:24:23.525 DEBUG:teuthology.provision.fog.smithi027:FOG request: POST /host/132/task {"taskTypeID": 1}
2023-08-16T04:24:23.542 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /tasktype {"name": "deploy"}
2023-08-16T04:24:23.552 DEBUG:teuthology.provision.fog.smithi142:FOG request: POST /host/247/task {"taskTypeID": 1}
2023-08-16T04:24:23.555 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None
2023-08-16T04:24:23.574 DEBUG:teuthology.provision.fog.smithi163:FOG request: POST /host/268/task {"taskTypeID": 1}
2023-08-16T04:24:23.582 DEBUG:teuthology.provision.fog.smithi105:FOG request: GET /task/active None
2023-08-16T04:24:23.589 DEBUG:teuthology.provision.fog.smithi027:FOG request: GET /task/active None
2023-08-16T04:24:23.631 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /task/active None
2023-08-16T04:24:23.638 DEBUG:teuthology.provision.fog.smithi142:FOG request: GET /task/active None
2023-08-16T04:24:23.683 INFO:teuthology.orchestra.console.smithi061:Power off
2023-08-16T04:24:23.684 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-08-16T04:24:23.707 INFO:teuthology.orchestra.console.smithi027:Power off
2023-08-16T04:24:23.708 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-08-16T04:24:23.726 INFO:teuthology.orchestra.console.smithi105:Power off
2023-08-16T04:24:23.727 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-08-16T04:24:23.735 DEBUG:teuthology.orchestra.console.smithi061:power off output: Chassis Power Control: Down/Off
2023-08-16T04:24:23.736 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:23.744 DEBUG:teuthology.orchestra.console.smithi027:power off output: Chassis Power Control: Down/Off
2023-08-16T04:24:23.745 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:23.762 INFO:teuthology.orchestra.console.smithi163:Power off
2023-08-16T04:24:23.763 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-08-16T04:24:23.777 INFO:teuthology.orchestra.console.smithi142:Power off
2023-08-16T04:24:23.778 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-08-16T04:24:23.787 DEBUG:teuthology.orchestra.console.smithi163:power off output: Chassis Power Control: Down/Off
2023-08-16T04:24:23.788 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:23.799 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on
2023-08-16T04:24:23.800 DEBUG:teuthology.orchestra.console.smithi105:power off output: Chassis Power Control: Down/Off
2023-08-16T04:24:23.800 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:23.809 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is on
2023-08-16T04:24:23.812 DEBUG:teuthology.orchestra.console.smithi142:power off output: Chassis Power Control: Down/Off
2023-08-16T04:24:23.812 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:23.823 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on
2023-08-16T04:24:23.824 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on
2023-08-16T04:24:23.835 DEBUG:teuthology.orchestra.console.smithi142:check power output: Chassis Power is on
2023-08-16T04:24:27.800 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:27.811 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:27.824 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:27.833 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:27.841 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:28.911 DEBUG:teuthology.orchestra.console.smithi027:check power output: 
2023-08-16T04:24:28.921 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2023-08-16T04:24:28.934 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-08-16T04:24:28.943 DEBUG:teuthology.orchestra.console.smithi163:check power output: 
2023-08-16T04:24:28.952 DEBUG:teuthology.orchestra.console.smithi142:check power output: 
2023-08-16T04:24:32.912 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:32.921 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:32.935 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:32.944 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:32.953 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:33.064 DEBUG:teuthology.orchestra.console.smithi142:check power output: Chassis Power is off
2023-08-16T04:24:33.165 INFO:teuthology.orchestra.console.smithi142:Power off completed
2023-08-16T04:24:33.267 INFO:teuthology.orchestra.console.smithi142:Power on
2023-08-16T04:24:33.267 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-08-16T04:24:33.288 DEBUG:teuthology.orchestra.console.smithi142:power on output: Chassis Power Control: Up/On
2023-08-16T04:24:33.289 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:33.310 DEBUG:teuthology.orchestra.console.smithi142:check power output: Chassis Power is off
2023-08-16T04:24:34.024 DEBUG:teuthology.orchestra.console.smithi027:check power output: 
2023-08-16T04:24:34.034 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2023-08-16T04:24:34.046 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-08-16T04:24:34.055 DEBUG:teuthology.orchestra.console.smithi163:check power output: 
2023-08-16T04:24:37.312 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:38.025 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:38.035 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:38.047 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:38.056 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:38.145 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is off
2023-08-16T04:24:38.247 INFO:teuthology.orchestra.console.smithi061:Power off completed
2023-08-16T04:24:38.348 INFO:teuthology.orchestra.console.smithi061:Power on
2023-08-16T04:24:38.348 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-08-16T04:24:38.368 DEBUG:teuthology.orchestra.console.smithi061:power on output: Chassis Power Control: Up/On
2023-08-16T04:24:38.369 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:38.391 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is off
2023-08-16T04:24:38.423 DEBUG:teuthology.orchestra.console.smithi142:check power output: 
2023-08-16T04:24:39.136 DEBUG:teuthology.orchestra.console.smithi027:check power output: 
2023-08-16T04:24:39.156 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-08-16T04:24:39.165 DEBUG:teuthology.orchestra.console.smithi163:check power output: 
2023-08-16T04:24:42.392 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:42.424 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:43.137 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:43.156 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:43.166 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:43.248 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is off
2023-08-16T04:24:43.277 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is off
2023-08-16T04:24:43.349 INFO:teuthology.orchestra.console.smithi027:Power off completed
2023-08-16T04:24:43.378 INFO:teuthology.orchestra.console.smithi163:Power off completed
2023-08-16T04:24:43.452 INFO:teuthology.orchestra.console.smithi027:Power on
2023-08-16T04:24:43.452 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-08-16T04:24:43.476 DEBUG:teuthology.orchestra.console.smithi027:power on output: Chassis Power Control: Up/On
2023-08-16T04:24:43.477 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:43.486 INFO:teuthology.orchestra.console.smithi163:Power on
2023-08-16T04:24:43.487 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-08-16T04:24:43.499 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is off
2023-08-16T04:24:43.502 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2023-08-16T04:24:43.509 DEBUG:teuthology.orchestra.console.smithi163:power on output: Chassis Power Control: Up/On
2023-08-16T04:24:43.509 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:43.531 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is off
2023-08-16T04:24:43.535 DEBUG:teuthology.orchestra.console.smithi142:check power output: 
2023-08-16T04:24:44.268 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-08-16T04:24:47.501 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:47.509 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:47.533 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:47.541 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:47.618 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is on
2023-08-16T04:24:47.720 INFO:teuthology.orchestra.console.smithi061:Power on completed
2023-08-16T04:24:47.820 INFO:teuthology.provision.fog.smithi061:Waiting for deploy to finish
2023-08-16T04:24:47.821 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None
2023-08-16T04:24:52.829 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:24:52.838 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on
2023-08-16T04:24:52.839 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on
2023-08-16T04:24:52.840 DEBUG:teuthology.orchestra.console.smithi142:check power output: Chassis Power is on
2023-08-16T04:24:52.940 INFO:teuthology.orchestra.console.smithi027:Power on completed
2023-08-16T04:24:52.940 INFO:teuthology.orchestra.console.smithi163:Power on completed
2023-08-16T04:24:52.940 INFO:teuthology.orchestra.console.smithi142:Power on completed
2023-08-16T04:24:53.041 INFO:teuthology.provision.fog.smithi027:Waiting for deploy to finish
2023-08-16T04:24:53.042 DEBUG:teuthology.provision.fog.smithi027:FOG request: GET /task/active None
2023-08-16T04:24:53.043 INFO:teuthology.provision.fog.smithi163:Waiting for deploy to finish
2023-08-16T04:24:53.043 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /task/active None
2023-08-16T04:24:53.044 INFO:teuthology.provision.fog.smithi142:Waiting for deploy to finish
2023-08-16T04:24:53.044 DEBUG:teuthology.provision.fog.smithi142:FOG request: GET /task/active None
2023-08-16T04:25:18.061 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None
2023-08-16T04:25:23.069 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is off
2023-08-16T04:25:23.170 INFO:teuthology.orchestra.console.smithi105:Power off completed
2023-08-16T04:25:23.271 INFO:teuthology.orchestra.console.smithi105:Power on
2023-08-16T04:25:23.271 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-08-16T04:25:23.295 DEBUG:teuthology.orchestra.console.smithi105:power on output: Chassis Power Control: Up/On
2023-08-16T04:25:23.295 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:25:23.317 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is off
2023-08-16T04:25:27.320 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:25:27.431 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is off
2023-08-16T04:25:31.436 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:25:32.547 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-08-16T04:25:36.548 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:25:37.659 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-08-16T04:25:38.072 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /task/active None
2023-08-16T04:25:38.075 DEBUG:teuthology.provision.fog.smithi027:FOG request: GET /task/active None
2023-08-16T04:25:41.366 DEBUG:teuthology.provision.fog.smithi142:FOG request: GET /task/active None
2023-08-16T04:25:41.367 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None
2023-08-16T04:25:41.660 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:25:42.773 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-08-16T04:25:46.776 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:25:47.887 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-08-16T04:25:51.892 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:25:53.003 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-08-16T04:25:56.440 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None
2023-08-16T04:26:01.391 DEBUG:teuthology.provision.fog.smithi027:FOG request: GET /task/active None
2023-08-16T04:26:01.393 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /task/active None
2023-08-16T04:26:01.394 DEBUG:teuthology.provision.fog.smithi142:FOG request: GET /task/active None
2023-08-16T04:26:01.395 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:26:02.504 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-08-16T04:26:06.508 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:26:07.618 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-08-16T04:26:11.620 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:26:12.732 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-08-16T04:26:16.484 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /task/active None
2023-08-16T04:26:16.486 DEBUG:teuthology.provision.fog.smithi142:FOG request: GET /task/active None
2023-08-16T04:26:16.490 DEBUG:teuthology.provision.fog.smithi027:FOG request: GET /task/active None
2023-08-16T04:26:16.491 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None
2023-08-16T04:26:16.733 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:26:17.845 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-08-16T04:26:21.848 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:26:22.959 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-08-16T04:26:26.960 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:26:28.071 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-08-16T04:26:31.577 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None
2023-08-16T04:26:31.579 DEBUG:teuthology.provision.fog.smithi142:FOG request: GET /task/active None
2023-08-16T04:26:31.580 DEBUG:teuthology.provision.fog.smithi027:FOG request: GET /task/active None
2023-08-16T04:26:31.581 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /task/active None
2023-08-16T04:26:51.603 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:26:52.714 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-08-16T04:26:56.716 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:26:57.827 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-08-16T04:26:57.828 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-08-16T04:26:57.828 DEBUG:teuthology.provision.fog.smithi105:Canceling deploy task with ID 442137
2023-08-16T04:26:57.828 DEBUG:teuthology.provision.fog.smithi105:FOG request: DELETE /task/cancel {"id": 442137}
2023-08-16T04:26:57.882 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi105
2023-08-16T04:26:57.883 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-08-16T04:26:58.022 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1de2052948b146aabd020728433922cf
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi105
2023-08-16T04:26:58.024 INFO:teuthology.nuke:Checking targets against current locks
2023-08-16T04:27:08.070 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /task/active None
2023-08-16T04:27:08.071 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None
2023-08-16T04:27:08.072 DEBUG:teuthology.provision.fog.smithi142:FOG request: GET /task/active None
2023-08-16T04:27:08.073 DEBUG:teuthology.provision.fog.smithi027:FOG request: GET /task/active None
2023-08-16T04:27:38.110 DEBUG:teuthology.nuke:shortname: smithi027
2023-08-16T04:27:38.112 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-08-16T04:27:38.139 DEBUG:teuthology.nuke:shortname: smithi061
2023-08-16T04:27:38.139 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-08-16T04:27:38.155 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-15_19:06:06-upgrade:octopus-x-wip-yuri8-testing-2023-08-11-0834-pacific-distro-default-smithi/7369497', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-16 04:23:50.737034', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='}
2023-08-16T04:27:38.179 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi061.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-15_19:06:06-upgrade:octopus-x-wip-yuri8-testing-2023-08-11-0834-pacific-distro-default-smithi/7369497', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-16 04:23:50.740260', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}
2023-08-16T04:27:38.193 DEBUG:teuthology.nuke:shortname: smithi105
2023-08-16T04:27:38.193 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-08-16T04:27:38.218 DEBUG:teuthology.nuke:shortname: smithi142
2023-08-16T04:27:38.219 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-08-16T04:27:38.227 INFO:teuthology.orchestra.console.smithi027:Power off
2023-08-16T04:27:38.228 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-08-16T04:27:38.244 DEBUG:teuthology.nuke:shortname: smithi163
2023-08-16T04:27:38.244 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-08-16T04:27:38.246 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-15_19:06:06-upgrade:octopus-x-wip-yuri8-testing-2023-08-11-0834-pacific-distro-default-smithi/7369497', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-16 04:23:50.738303', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}
2023-08-16T04:27:38.253 DEBUG:teuthology.orchestra.console.smithi027:power off output: Chassis Power Control: Down/Off
2023-08-16T04:27:38.254 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:27:38.265 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi142.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-15_19:06:06-upgrade:octopus-x-wip-yuri8-testing-2023-08-11-0834-pacific-distro-default-smithi/7369497', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-16 04:23:50.741323', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC014sjPsv3z7URnzBMvnBgjfZqWWAIhmz6JqsFfU21tuUEj0mAwd91Hbt3PuVavovKmM/+oHERPJ+T/Wr8i6gbwlnT0YA8f2Fvu07lsXMyL8Ao+XbZ13NpvbGcdfbkGkko+l6lqCFsyplA8y9aadyDL5SyQ+MWXtWeSH5I3j19Z3/TdgWT9wB6kICDt27V4LXx8ONwaYwcF5F+4ej5e6zXeK7sWXhsXhvh0OOUGJ/DfoUlALxnTyF+437c1SjpJz9e2M+cwxqK/qu47zE1ltgq5F2cY1VRKLumENeU8sVcq9RYTBzvqdIW1/QndbJ6Qv0MVdxY17h8crPpFggTOo/hfs5uiIBybdZeP32iVOxpqD5IG3zAspfjPyYYVrIS1oZ6RIZMF0V1OZlfI3Z4KNBJV5+4oIICH+KqGXeqTAg1952tvQVI/J8HEiQgTj682ftdMX2G/NEfvxWUAZVy7rP5qMcYUFPMXXXPdBGNqpK+eC0i0z9AwVSpka6mI47ZTDc='}
2023-08-16T04:27:38.277 INFO:teuthology.orchestra.console.smithi061:Power off
2023-08-16T04:27:38.278 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-08-16T04:27:38.287 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-15_19:06:06-upgrade:octopus-x-wip-yuri8-testing-2023-08-11-0834-pacific-distro-default-smithi/7369497', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-16 04:23:50.739281', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCa7N4rN+rDOiIATEqkWRXzZxNiVvriCyKV/SG/8HMPv8bZcD/9HlQBmytd9QkcGcRfvyj01S4MijEzZdMzabmrX6mBMYB1kldGiW/SJBd+jtyeZyWXCj1M15aUDcF/3u5qga3EC5LfDMfcOKs2x64cdscXK4w8IcY8MtZuuxWSF97p4gFe3zJ8qLTCRVWYwzQ5pUXKppltUl6w5VUb/i2+CSgtlFfnMTlkI9owZ93hWrFXz3847uXrzLPI4kW3GI4sUKI6jNss/uwKw268zTaN/i8Kw1cpcnEF2BFHf4YAFQ8vbuYppAQz6ts7Jq4XHvCQpHwcqwMLu/NciYFlFcrPI2EqUD4MvBeg7r5SObD0keRI6hTx8MqRJqSMLxbfFiGJFXEtY/ljpZvAyMfugzhWT0bmpI4jOJ1IEFac+BEEu3hwuNj65ZX4E3r/0q8uSNpvie/+96ioJGZnB5nkXMrz3g8MpBUV9ro0dO84q3hczcPi5qe3B385R6Hb9UGHlTc='}
2023-08-16T04:27:38.293 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on
2023-08-16T04:27:38.299 DEBUG:teuthology.orchestra.console.smithi061:power off output: Chassis Power Control: Down/Off
2023-08-16T04:27:38.300 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:27:38.321 INFO:teuthology.orchestra.console.smithi105:Power off
2023-08-16T04:27:38.322 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-08-16T04:27:38.335 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is on
2023-08-16T04:27:38.344 DEBUG:teuthology.orchestra.console.smithi105:power off output: Chassis Power Control: Down/Off
2023-08-16T04:27:38.344 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:27:38.367 INFO:teuthology.orchestra.console.smithi163:Power off
2023-08-16T04:27:38.368 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-08-16T04:27:38.391 INFO:teuthology.orchestra.console.smithi142:Power off
2023-08-16T04:27:38.393 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-08-16T04:27:38.403 DEBUG:teuthology.orchestra.console.smithi163:power off output: Chassis Power Control: Down/Off
2023-08-16T04:27:38.405 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:27:38.415 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on
2023-08-16T04:27:39.416 DEBUG:teuthology.orchestra.console.smithi163:check power output: 
2023-08-16T04:27:42.298 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:27:42.336 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:27:42.418 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:27:43.409 DEBUG:teuthology.orchestra.console.smithi027:check power output: 
2023-08-16T04:27:43.417 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:27:43.448 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2023-08-16T04:27:43.530 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-08-16T04:27:44.531 DEBUG:teuthology.orchestra.console.smithi163:check power output: 
2023-08-16T04:27:47.411 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:27:47.449 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:27:47.531 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:27:48.433 DEBUG:teuthology.orchestra.console.smithi142:power off output: Chassis Power Control: Down/Off
2023-08-16T04:27:48.434 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:27:48.523 DEBUG:teuthology.orchestra.console.smithi027:check power output: 
2023-08-16T04:27:48.532 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:27:48.560 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2023-08-16T04:27:48.644 DEBUG:teuthology.orchestra.console.smithi105:check power output: 
2023-08-16T04:27:49.445 DEBUG:teuthology.orchestra.console.smithi142:check power output: 
2023-08-16T04:27:49.643 DEBUG:teuthology.orchestra.console.smithi163:check power output: 
2023-08-16T04:27:52.526 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:27:52.562 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:27:52.645 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:27:53.110 DEBUG:teuthology.provision.fog.smithi142:FOG request: GET /task/active None
2023-08-16T04:27:53.113 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None
2023-08-16T04:27:53.115 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /task/active None
2023-08-16T04:28:08.134 DEBUG:teuthology.provision.fog.smithi027:FOG request: GET /task/active None
2023-08-16T04:28:08.135 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:28:08.143 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-16T04:28:13.156 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is off
2023-08-16T04:28:13.157 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is off
2023-08-16T04:28:13.157 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is off
2023-08-16T04:28:13.163 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-16T04:28:13.167 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-16T04:28:13.169 DEBUG:teuthology.orchestra.console.smithi142:check power output: Chassis Power is off
2023-08-16T04:28:13.170 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is off
2023-08-16T04:28:13.257 INFO:teuthology.orchestra.console.smithi027:Power off completed
2023-08-16T04:28:13.258 INFO:teuthology.orchestra.console.smithi061:Power off completed
2023-08-16T04:28:13.259 INFO:teuthology.orchestra.console.smithi105:Power off completed
2023-08-16T04:28:13.271 INFO:teuthology.orchestra.console.smithi142:Power off completed
2023-08-16T04:28:13.272 INFO:teuthology.orchestra.console.smithi163:Power off completed
2023-08-16T04:28:24.175 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61
2023-08-16T04:28:24.176 WARNING:teuthology.provision.fog.smithi142:[Errno None] Unable to connect to port 22 on 172.21.15.142
2023-08-16T04:28:34.202 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /task/active None
2023-08-16T04:28:34.204 DEBUG:teuthology.provision.fog.smithi027:FOG request: GET /task/active None
2023-08-16T04:28:34.205 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-16T04:28:34.209 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-16T04:28:39.853 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61
2023-08-16T04:28:39.853 WARNING:teuthology.provision.fog.smithi142:[Errno None] Unable to connect to port 22 on 172.21.15.142
2023-08-16T04:28:44.858 INFO:teuthology.lock.ops:unlocked: smithi061.front.sepia.ceph.com
2023-08-16T04:28:44.858 INFO:teuthology.lock.ops:unlocked: smithi027.front.sepia.ceph.com
2023-08-16T04:28:44.891 INFO:teuthology.lock.ops:unlocked: smithi163.front.sepia.ceph.com
2023-08-16T04:28:44.900 INFO:teuthology.lock.ops:unlocked: smithi105.front.sepia.ceph.com
2023-08-16T04:28:44.908 INFO:teuthology.lock.ops:unlocked: smithi142.front.sepia.ceph.com
2023-08-16T04:28:45.420 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi105