2023-09-21T12:23:55.172 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-21T12:23:55.174 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-09-21T12:23:55.357 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f2a17413ca0>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f2a170c4af0>, signals=[15])
2023-09-21T12:23:55.358 INFO:teuthology.lock.ops:Start node 'smithi084.front.sepia.ceph.com' reimaging
2023-09-21T12:23:55.359 INFO:teuthology.lock.ops:Updating [smithi084.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T12:23:55.359 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server
2023-09-21T12:23:55.377 INFO:teuthology.lock.ops:Start node 'smithi099.front.sepia.ceph.com' reimaging
2023-09-21T12:23:55.378 INFO:teuthology.lock.ops:Updating [smithi099.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T12:23:55.378 INFO:teuthology.lock.ops:Updating smithi099.front.sepia.ceph.com on lock server
2023-09-21T12:23:55.379 DEBUG:teuthology.provision.fog.smithi084:FOG request: GET /host {"name": "smithi084"}
2023-09-21T12:23:55.402 INFO:teuthology.lock.ops:Start node 'smithi112.front.sepia.ceph.com' reimaging
2023-09-21T12:23:55.403 INFO:teuthology.lock.ops:Updating [smithi112.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T12:23:55.403 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server
2023-09-21T12:23:55.404 DEBUG:teuthology.provision.fog.smithi099:FOG request: GET /host {"name": "smithi099"}
2023-09-21T12:23:55.427 DEBUG:teuthology.provision.fog.smithi112:FOG request: GET /host {"name": "smithi112"}
2023-09-21T12:23:55.456 DEBUG:teuthology.provision.fog.smithi084:FOG request: GET /image {"name": "smithi_centos_8.stream"}
2023-09-21T12:23:55.469 DEBUG:teuthology.provision.fog.smithi099:FOG request: GET /image {"name": "smithi_centos_8.stream"}
2023-09-21T12:23:55.562 DEBUG:teuthology.provision.fog.smithi084:Requesting image smithi_centos_8.stream (ID 3)
2023-09-21T12:23:55.563 DEBUG:teuthology.provision.fog.smithi084:FOG request: PUT /host/189 {"imageID": 3}
2023-09-21T12:23:55.592 DEBUG:teuthology.provision.fog.smithi112:FOG request: GET /image {"name": "smithi_centos_8.stream"}
2023-09-21T12:23:55.609 INFO:teuthology.provision.fog.smithi084:Scheduling deploy of centos 8.stream
2023-09-21T12:23:55.609 DEBUG:teuthology.provision.fog.smithi084:FOG request: GET /task/active None
2023-09-21T12:23:55.613 DEBUG:teuthology.provision.fog.smithi099:Requesting image smithi_centos_8.stream (ID 3)
2023-09-21T12:23:55.614 DEBUG:teuthology.provision.fog.smithi099:FOG request: PUT /host/204 {"imageID": 3}
2023-09-21T12:23:55.643 DEBUG:teuthology.provision.fog.smithi112:Requesting image smithi_centos_8.stream (ID 3)
2023-09-21T12:23:55.643 DEBUG:teuthology.provision.fog.smithi112:FOG request: PUT /host/217 {"imageID": 3}
2023-09-21T12:23:55.660 INFO:teuthology.provision.fog.smithi099:Scheduling deploy of centos 8.stream
2023-09-21T12:23:55.660 DEBUG:teuthology.provision.fog.smithi099:FOG request: GET /task/active None
2023-09-21T12:23:55.691 DEBUG:teuthology.provision.fog.smithi084:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T12:23:55.694 INFO:teuthology.provision.fog.smithi112:Scheduling deploy of centos 8.stream
2023-09-21T12:23:55.694 DEBUG:teuthology.provision.fog.smithi112:FOG request: GET /task/active None
2023-09-21T12:23:55.738 DEBUG:teuthology.provision.fog.smithi084:FOG request: POST /host/189/task {"taskTypeID": 1}
2023-09-21T12:23:55.743 DEBUG:teuthology.provision.fog.smithi099:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T12:23:55.768 DEBUG:teuthology.provision.fog.smithi112:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T12:23:55.785 DEBUG:teuthology.provision.fog.smithi099:FOG request: POST /host/204/task {"taskTypeID": 1}
2023-09-21T12:23:55.813 DEBUG:teuthology.provision.fog.smithi112:FOG request: POST /host/217/task {"taskTypeID": 1}
2023-09-21T12:23:55.821 DEBUG:teuthology.provision.fog.smithi084:FOG request: GET /task/active None
2023-09-21T12:23:55.833 DEBUG:teuthology.provision.fog.smithi099:FOG request: GET /task/active None
2023-09-21T12:23:55.882 DEBUG:teuthology.provision.fog.smithi112:FOG request: GET /task/active None
2023-09-21T12:23:55.939 INFO:teuthology.orchestra.console.smithi099:Power off
2023-09-21T12:23:55.940 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:23:55.955 INFO:teuthology.orchestra.console.smithi084:Power off
2023-09-21T12:23:55.955 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:23:55.977 DEBUG:teuthology.orchestra.console.smithi084:power off output: Chassis Power Control: Down/Off
2023-09-21T12:23:55.978 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:23:55.996 INFO:teuthology.orchestra.console.smithi112:Power off
2023-09-21T12:23:55.996 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:23:56.005 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is off
2023-09-21T12:23:56.106 INFO:teuthology.orchestra.console.smithi084:Power off completed
2023-09-21T12:23:56.206 INFO:teuthology.orchestra.console.smithi084:Power on
2023-09-21T12:23:56.268 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T12:23:56.288 DEBUG:teuthology.orchestra.console.smithi084:power on output: Chassis Power Control: Up/On
2023-09-21T12:23:56.289 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:23:56.309 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is off
2023-09-21T12:24:00.314 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:24:00.422 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is off
2023-09-21T12:24:04.426 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:24:04.534 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on
2023-09-21T12:24:04.635 INFO:teuthology.orchestra.console.smithi084:Power on completed
2023-09-21T12:24:04.736 INFO:teuthology.provision.fog.smithi084:Waiting for deploy to finish
2023-09-21T12:24:04.736 DEBUG:teuthology.provision.fog.smithi084:FOG request: GET /task/active None
2023-09-21T12:24:16.011 DEBUG:teuthology.orchestra.console.smithi099:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:24:16.012 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:24:16.064 DEBUG:teuthology.orchestra.console.smithi112:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:24:16.064 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:24:19.822 DEBUG:teuthology.provision.fog.smithi084:FOG request: GET /task/active None
2023-09-21T12:24:34.910 DEBUG:teuthology.provision.fog.smithi084:FOG request: GET /task/active None
2023-09-21T12:24:36.086 DEBUG:teuthology.orchestra.console.smithi099:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:24:36.086 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:24:36.134 DEBUG:teuthology.orchestra.console.smithi112:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:24:36.134 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:24:49.988 DEBUG:teuthology.provision.fog.smithi084:FOG request: GET /task/active None
2023-09-21T12:24:56.155 DEBUG:teuthology.orchestra.console.smithi099:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:24:56.156 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:24:56.204 DEBUG:teuthology.orchestra.console.smithi112:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:24:56.205 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:24:57.164 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:24:57.213 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:25:01.170 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:25:01.214 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:25:02.280 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:25:02.325 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:25:05.059 DEBUG:teuthology.provision.fog.smithi084:FOG request: GET /task/active None
2023-09-21T12:25:06.282 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:25:06.326 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:25:07.391 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:25:07.434 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:25:11.396 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:25:11.436 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:25:12.506 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:25:12.547 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:25:16.510 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:25:16.548 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:25:17.621 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:25:17.658 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:25:20.133 DEBUG:teuthology.provision.fog.smithi084:FOG request: GET /task/active None
2023-09-21T12:25:21.623 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:25:21.659 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:25:22.732 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:25:22.768 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:25:26.738 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:25:26.770 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:25:27.847 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:25:27.878 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:25:31.850 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:25:31.879 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:25:32.961 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:25:32.987 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:25:35.206 DEBUG:teuthology.provision.fog.smithi084:FOG request: GET /task/active None
2023-09-21T12:25:36.963 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:25:36.988 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:25:38.071 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:25:38.097 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:25:42.074 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:25:42.099 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:25:43.184 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:25:43.209 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:25:47.186 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:25:47.209 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:25:48.294 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:25:48.322 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:25:50.270 DEBUG:teuthology.provision.fog.smithi084:FOG request: GET /task/active None
2023-09-21T12:25:52.298 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:25:52.324 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:25:53.408 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:25:53.435 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:25:57.414 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:25:57.437 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:25:58.524 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:25:58.545 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:26:02.530 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:26:02.547 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:26:03.640 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:26:03.656 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:26:05.369 DEBUG:teuthology.provision.fog.smithi084:FOG request: GET /task/active None
2023-09-21T12:26:07.644 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:26:07.725 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:26:08.828 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:26:08.829 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T12:26:08.829 ERROR:teuthology.orchestra.console.smithi099:Failed to power off
2023-09-21T12:26:08.830 INFO:teuthology.orchestra.console.smithi099:Power on
2023-09-21T12:26:08.830 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T12:26:08.837 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:26:08.838 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T12:26:08.838 ERROR:teuthology.orchestra.console.smithi112:Failed to power off
2023-09-21T12:26:08.838 INFO:teuthology.orchestra.console.smithi112:Power on
2023-09-21T12:26:08.839 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T12:26:20.454 DEBUG:teuthology.provision.fog.smithi084:FOG request: GET /task/active None
2023-09-21T12:26:28.903 DEBUG:teuthology.orchestra.console.smithi112:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:26:28.904 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T12:26:28.911 DEBUG:teuthology.orchestra.console.smithi099:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:26:28.912 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T12:26:35.534 DEBUG:teuthology.provision.fog.smithi084:FOG request: GET /task/active None
2023-09-21T12:26:35.615 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:26:48.972 DEBUG:teuthology.orchestra.console.smithi112:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:26:48.972 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T12:26:48.982 DEBUG:teuthology.orchestra.console.smithi099:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:26:48.983 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T12:26:53.978 WARNING:teuthology.provision.fog.smithi084:[Errno None] Unable to connect to port 22 on 172.21.15.84
2023-09-21T12:26:59.984 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:27:03.070 WARNING:teuthology.provision.fog.smithi084:[Errno None] Unable to connect to port 22 on 172.21.15.84
2023-09-21T12:27:09.047 DEBUG:teuthology.orchestra.console.smithi112:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:27:09.048 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:27:09.057 DEBUG:teuthology.orchestra.console.smithi099:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:27:09.057 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:27:09.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:27:10.057 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:27:10.063 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:27:12.158 WARNING:teuthology.provision.fog.smithi084:[Errno None] Unable to connect to port 22 on 172.21.15.84
2023-09-21T12:27:14.061 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:27:14.068 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:27:15.170 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:27:15.178 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:27:18.162 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:27:19.173 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:27:19.180 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:27:20.284 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:27:20.288 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:27:21.243 WARNING:teuthology.provision.fog.smithi084:[Errno None] Unable to connect to port 22 on 172.21.15.84
2023-09-21T12:27:24.286 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:27:24.292 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:27:25.395 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:27:25.440 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:27:27.246 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:27:29.442 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:27:29.449 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:27:30.302 WARNING:teuthology.provision.fog.smithi084:[Errno None] Unable to connect to port 22 on 172.21.15.84
2023-09-21T12:27:30.550 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:27:30.557 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:27:34.556 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:27:34.563 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:27:35.665 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:27:35.671 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:27:36.305 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:27:39.673 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:27:39.684 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:27:40.785 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:27:40.793 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:27:44.790 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:27:44.796 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:27:45.899 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:27:45.905 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:27:49.902 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:27:49.909 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:27:51.011 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:27:51.017 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:27:55.014 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:27:55.021 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:27:56.124 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:27:56.129 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:28:00.131 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:28:00.139 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:28:01.242 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:28:01.248 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:28:05.246 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:28:05.253 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:28:06.356 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:28:06.365 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:28:07.906 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-09-21T12:28:07.907 DEBUG:teuthology.orchestra.run.smithi084:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-09-21T12:28:10.357 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:28:10.366 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:28:11.466 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:28:11.474 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:28:15.467 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:28:15.475 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:28:16.577 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:28:16.583 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:28:20.579 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:28:20.586 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:28:21.688 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:28:21.689 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T12:28:21.689 DEBUG:teuthology.provision.fog.smithi112:Canceling deploy task with ID 499521
2023-09-21T12:28:21.689 DEBUG:teuthology.provision.fog.smithi112:FOG request: DELETE /task/cancel {"id": 499521}
2023-09-21T12:28:21.694 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:28:21.695 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T12:28:21.695 DEBUG:teuthology.provision.fog.smithi099:Canceling deploy task with ID 499520
2023-09-21T12:28:21.695 DEBUG:teuthology.provision.fog.smithi099:FOG request: DELETE /task/cancel {"id": 499520}
2023-09-21T12:28:21.748 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi099
2023-09-21T12:28:21.749 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-09-21T12:28:21.911 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ef7d5ae023a04d999a70f39ad76541d5
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi099
2023-09-21T12:28:21.913 INFO:teuthology.nuke:Checking targets against current locks
2023-09-21T12:28:21.984 DEBUG:teuthology.nuke:shortname: smithi084
2023-09-21T12:28:22.008 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T12:28:22.045 DEBUG:teuthology.nuke:shortname: smithi099
2023-09-21T12:28:22.046 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T12:28:22.051 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi084.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-09-20_16:32:45-fs-wip-lusov-config-format-distro-default-smithi/7399865', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 12:23:54.011631', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBgG75AlOnTvqTQSvIYiBLjh2GTidAVlz7LwITXHnMsjgilP5O/QFlxcgCJp5WBljsXWCOxVrB5RvpWsEfi0/d8='}
2023-09-21T12:28:22.055 DEBUG:teuthology.nuke:shortname: smithi112
2023-09-21T12:28:22.055 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T12:28:22.066 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi099.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-09-20_16:32:45-fs-wip-lusov-config-format-distro-default-smithi/7399865', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 12:23:54.009818', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFXRHbCQKDEuyNLFw4V1AtfgTXVj1Soceqgh+qxIZuwuU9XS7E6mMjA4WiDriC/6XhxlgPBiXLgPCx7jyVR8EVs='}
2023-09-21T12:28:22.078 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-09-20_16:32:45-fs-wip-lusov-config-format-distro-default-smithi/7399865', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 12:23:54.012736', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNsN1SFtSMFYxyYNcbgvW1eDZSesLM9jY4HJEMXKIWGOAdf0ieWEX3W/j2HEYjhwPyOGTJiDQ1tGjXWJ9ToC5jc='}
2023-09-21T12:28:22.110 INFO:teuthology.orchestra.console.smithi084:Power off
2023-09-21T12:28:22.111 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:28:22.125 INFO:teuthology.orchestra.console.smithi099:Power off
2023-09-21T12:28:22.127 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:28:22.140 DEBUG:teuthology.orchestra.console.smithi084:power off output: Chassis Power Control: Down/Off
2023-09-21T12:28:22.141 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:28:22.166 INFO:teuthology.orchestra.console.smithi112:Power off
2023-09-21T12:28:22.168 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:28:22.177 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on
2023-09-21T12:28:25.801 DEBUG:teuthology.orchestra.run:got remote process result: None
2023-09-21T12:28:26.180 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:28:26.290 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on
2023-09-21T12:28:30.294 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:28:30.404 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on
2023-09-21T12:28:34.406 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:28:34.519 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is off
2023-09-21T12:28:34.620 INFO:teuthology.orchestra.console.smithi084:Power off completed
2023-09-21T12:28:34.775 INFO:teuthology.lock.ops:unlocked: smithi084.front.sepia.ceph.com
2023-09-21T12:28:42.200 DEBUG:teuthology.orchestra.console.smithi099:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:28:42.201 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:28:42.235 DEBUG:teuthology.orchestra.console.smithi112:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:28:42.236 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:29:02.277 DEBUG:teuthology.orchestra.console.smithi099:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:29:02.278 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:29:02.312 DEBUG:teuthology.orchestra.console.smithi112:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:29:02.312 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:29:22.349 DEBUG:teuthology.orchestra.console.smithi099:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:29:22.350 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:29:22.381 DEBUG:teuthology.orchestra.console.smithi112:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:29:22.382 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:29:23.360 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:29:23.391 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:29:27.362 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:29:27.392 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:29:28.473 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:29:28.501 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:29:32.478 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:29:32.503 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:29:33.589 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:29:33.612 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:29:37.594 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:29:37.613 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:29:38.704 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:29:38.722 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:29:42.706 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:29:42.723 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:29:43.817 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:29:43.832 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:29:47.823 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:29:47.833 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:29:48.934 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:29:48.943 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:29:52.938 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:29:52.946 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:29:54.050 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:29:54.056 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:29:58.055 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:29:58.063 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:29:59.166 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:29:59.172 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:30:03.170 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:30:03.178 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:30:04.281 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:30:04.288 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:30:08.286 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:30:08.294 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:30:09.397 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:30:09.403 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:30:13.402 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:30:13.409 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:30:14.512 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:30:14.518 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:30:18.518 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:30:18.525 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:30:19.629 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:30:19.635 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:30:23.634 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:30:23.643 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:30:24.746 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:30:24.752 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:30:28.750 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:30:28.758 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:30:29.860 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:30:29.868 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:30:33.866 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:30:33.873 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:30:34.975 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T12:30:34.976 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T12:30:34.977 ERROR:teuthology.orchestra.console.smithi099:Failed to power off
2023-09-21T12:30:34.981 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:30:34.982 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T12:30:34.982 ERROR:teuthology.orchestra.console.smithi112:Failed to power off
2023-09-21T12:30:35.130 INFO:teuthology.lock.ops:unlocked: smithi112.front.sepia.ceph.com
2023-09-21T12:30:35.138 INFO:teuthology.lock.ops:unlocked: smithi099.front.sepia.ceph.com
2023-09-21T12:30:35.491 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi099