2023-11-04T04:48:44.053 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2
2023-11-04T04:48:44.055 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T04:48:44.228 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f1166ffab20>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f1166cafaf0>, signals=[15])
2023-11-04T04:48:44.229 INFO:teuthology.lock.ops:Start node 'smithi012.front.sepia.ceph.com' reimaging
2023-11-04T04:48:44.229 INFO:teuthology.lock.ops:Updating [smithi012.front.sepia.ceph.com]: reset os type and version on server
2023-11-04T04:48:44.230 INFO:teuthology.lock.ops:Updating smithi012.front.sepia.ceph.com on lock server
2023-11-04T04:48:44.249 INFO:teuthology.lock.ops:Start node 'smithi064.front.sepia.ceph.com' reimaging
2023-11-04T04:48:44.250 INFO:teuthology.lock.ops:Updating [smithi064.front.sepia.ceph.com]: reset os type and version on server
2023-11-04T04:48:44.250 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server
2023-11-04T04:48:44.251 DEBUG:teuthology.provision.fog.smithi012:FOG request: GET /host {"name": "smithi012"}
2023-11-04T04:48:44.274 INFO:teuthology.lock.ops:Start node 'smithi107.front.sepia.ceph.com' reimaging
2023-11-04T04:48:44.274 INFO:teuthology.lock.ops:Updating [smithi107.front.sepia.ceph.com]: reset os type and version on server
2023-11-04T04:48:44.275 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server
2023-11-04T04:48:44.276 DEBUG:teuthology.provision.fog.smithi064:FOG request: GET /host {"name": "smithi064"}
2023-11-04T04:48:44.301 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /host {"name": "smithi107"}
2023-11-04T04:48:44.320 DEBUG:teuthology.provision.fog.smithi012:FOG request: GET /image {"name": "smithi_centos_8.stream"}
2023-11-04T04:48:44.367 DEBUG:teuthology.provision.fog.smithi064:FOG request: GET /image {"name": "smithi_centos_8.stream"}
2023-11-04T04:48:44.374 DEBUG:teuthology.provision.fog.smithi012:Requesting image smithi_centos_8.stream (ID 3)
2023-11-04T04:48:44.374 DEBUG:teuthology.provision.fog.smithi012:FOG request: PUT /host/118 {"imageID": 3}
2023-11-04T04:48:44.378 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /image {"name": "smithi_centos_8.stream"}
2023-11-04T04:48:44.408 DEBUG:teuthology.provision.fog.smithi064:Requesting image smithi_centos_8.stream (ID 3)
2023-11-04T04:48:44.409 DEBUG:teuthology.provision.fog.smithi064:FOG request: PUT /host/169 {"imageID": 3}
2023-11-04T04:48:44.412 INFO:teuthology.provision.fog.smithi012:Scheduling deploy of centos 8.stream
2023-11-04T04:48:44.412 DEBUG:teuthology.provision.fog.smithi012:FOG request: GET /task/active None
2023-11-04T04:48:44.420 DEBUG:teuthology.provision.fog.smithi107:Requesting image smithi_centos_8.stream (ID 3)
2023-11-04T04:48:44.420 DEBUG:teuthology.provision.fog.smithi107:FOG request: PUT /host/212 {"imageID": 3}
2023-11-04T04:48:44.453 INFO:teuthology.provision.fog.smithi064:Scheduling deploy of centos 8.stream
2023-11-04T04:48:44.454 DEBUG:teuthology.provision.fog.smithi064:FOG request: GET /task/active None
2023-11-04T04:48:44.468 INFO:teuthology.provision.fog.smithi107:Scheduling deploy of centos 8.stream
2023-11-04T04:48:44.468 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /task/active None
2023-11-04T04:48:44.476 DEBUG:teuthology.provision.fog.smithi012:FOG request: GET /tasktype {"name": "deploy"}
2023-11-04T04:48:44.508 DEBUG:teuthology.provision.fog.smithi012:FOG request: POST /host/118/task {"taskTypeID": 1}
2023-11-04T04:48:44.513 DEBUG:teuthology.provision.fog.smithi064:FOG request: GET /tasktype {"name": "deploy"}
2023-11-04T04:48:44.548 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /tasktype {"name": "deploy"}
2023-11-04T04:48:44.562 DEBUG:teuthology.provision.fog.smithi064:FOG request: POST /host/169/task {"taskTypeID": 1}
2023-11-04T04:48:44.588 DEBUG:teuthology.provision.fog.smithi012:FOG request: GET /task/active None
2023-11-04T04:48:44.589 DEBUG:teuthology.provision.fog.smithi107:FOG request: POST /host/212/task {"taskTypeID": 1}
2023-11-04T04:48:44.615 DEBUG:teuthology.provision.fog.smithi064:FOG request: GET /task/active None
2023-11-04T04:48:44.644 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /task/active None
2023-11-04T04:48:44.707 INFO:teuthology.orchestra.console.smithi012:Power off
2023-11-04T04:48:44.707 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:48:44.729 DEBUG:teuthology.orchestra.console.smithi012:power off output: Chassis Power Control: Down/Off
2023-11-04T04:48:44.729 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:48:44.743 INFO:teuthology.orchestra.console.smithi064:Power off
2023-11-04T04:48:44.744 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:48:44.757 INFO:teuthology.orchestra.console.smithi107:Power off
2023-11-04T04:48:44.758 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:48:44.765 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on
2023-11-04T04:48:44.766 DEBUG:teuthology.orchestra.console.smithi064:power off output: Chassis Power Control: Down/Off
2023-11-04T04:48:44.766 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:48:44.788 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on
2023-11-04T04:48:48.770 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:48:48.789 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:48:48.881 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on
2023-11-04T04:48:48.900 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on
2023-11-04T04:48:52.886 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:48:52.900 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:48:52.996 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is off
2023-11-04T04:48:53.009 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is off
2023-11-04T04:48:53.098 INFO:teuthology.orchestra.console.smithi012:Power off completed
2023-11-04T04:48:53.110 INFO:teuthology.orchestra.console.smithi064:Power off completed
2023-11-04T04:48:53.199 INFO:teuthology.orchestra.console.smithi012:Power on
2023-11-04T04:48:53.200 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T04:48:53.211 INFO:teuthology.orchestra.console.smithi064:Power on
2023-11-04T04:48:53.212 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T04:48:53.225 DEBUG:teuthology.orchestra.console.smithi012:power on output: Chassis Power Control: Up/On
2023-11-04T04:48:53.225 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:48:53.233 DEBUG:teuthology.orchestra.console.smithi064:power on output: Chassis Power Control: Up/On
2023-11-04T04:48:53.234 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:48:53.248 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is off
2023-11-04T04:48:53.256 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is off
2023-11-04T04:48:57.251 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:48:57.259 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:48:57.362 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is off
2023-11-04T04:48:57.369 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is off
2023-11-04T04:49:01.366 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:49:01.374 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:49:01.476 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on
2023-11-04T04:49:01.483 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on
2023-11-04T04:49:01.578 INFO:teuthology.orchestra.console.smithi012:Power on completed
2023-11-04T04:49:01.585 INFO:teuthology.orchestra.console.smithi064:Power on completed
2023-11-04T04:49:01.679 INFO:teuthology.provision.fog.smithi012:Waiting for deploy to finish
2023-11-04T04:49:01.680 DEBUG:teuthology.provision.fog.smithi012:FOG request: GET /task/active None
2023-11-04T04:49:01.685 INFO:teuthology.provision.fog.smithi064:Waiting for deploy to finish
2023-11-04T04:49:01.686 DEBUG:teuthology.provision.fog.smithi064:FOG request: GET /task/active None
2023-11-04T04:49:04.824 DEBUG:teuthology.orchestra.console.smithi107:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:49:04.824 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:49:16.799 DEBUG:teuthology.provision.fog.smithi012:FOG request: GET /task/active None
2023-11-04T04:49:16.801 DEBUG:teuthology.provision.fog.smithi064:FOG request: GET /task/active None
2023-11-04T04:49:24.893 DEBUG:teuthology.orchestra.console.smithi107:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:49:24.894 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:49:31.885 DEBUG:teuthology.provision.fog.smithi012:FOG request: GET /task/active None
2023-11-04T04:49:31.888 DEBUG:teuthology.provision.fog.smithi064:FOG request: GET /task/active None
2023-11-04T04:49:44.962 DEBUG:teuthology.orchestra.console.smithi107:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:49:44.963 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:49:45.972 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:49:46.966 DEBUG:teuthology.provision.fog.smithi012:FOG request: GET /task/active None
2023-11-04T04:49:46.975 DEBUG:teuthology.provision.fog.smithi064:FOG request: GET /task/active None
2023-11-04T04:49:49.974 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:49:51.084 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:49:55.087 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:49:56.196 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:50:00.200 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:50:01.310 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:50:02.056 DEBUG:teuthology.provision.fog.smithi012:FOG request: GET /task/active None
2023-11-04T04:50:02.058 DEBUG:teuthology.provision.fog.smithi064:FOG request: GET /task/active None
2023-11-04T04:50:05.314 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:50:06.425 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:50:10.426 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:50:11.534 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:50:15.538 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:50:16.646 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:50:17.136 DEBUG:teuthology.provision.fog.smithi012:FOG request: GET /task/active None
2023-11-04T04:50:17.140 DEBUG:teuthology.provision.fog.smithi064:FOG request: GET /task/active None
2023-11-04T04:50:20.650 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:50:21.761 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:50:25.765 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:50:26.874 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:50:30.878 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:50:31.986 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:50:32.219 DEBUG:teuthology.provision.fog.smithi064:FOG request: GET /task/active None
2023-11-04T04:50:32.224 DEBUG:teuthology.provision.fog.smithi012:FOG request: GET /task/active None
2023-11-04T04:50:35.990 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:50:37.099 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:50:41.102 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:50:42.213 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:50:46.218 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:50:47.300 DEBUG:teuthology.provision.fog.smithi012:FOG request: GET /task/active None
2023-11-04T04:50:47.304 DEBUG:teuthology.provision.fog.smithi064:FOG request: GET /task/active None
2023-11-04T04:50:47.329 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:50:51.334 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:50:52.442 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:50:56.446 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:50:57.557 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:50:57.558 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T04:50:57.558 ERROR:teuthology.orchestra.console.smithi107:Failed to power off
2023-11-04T04:50:57.559 INFO:teuthology.orchestra.console.smithi107:Power on
2023-11-04T04:50:57.559 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T04:51:02.378 DEBUG:teuthology.provision.fog.smithi064:FOG request: GET /task/active None
2023-11-04T04:51:02.380 DEBUG:teuthology.provision.fog.smithi012:FOG request: GET /task/active None
2023-11-04T04:51:17.466 DEBUG:teuthology.provision.fog.smithi012:FOG request: GET /task/active None
2023-11-04T04:51:17.468 DEBUG:teuthology.provision.fog.smithi064:FOG request: GET /task/active None
2023-11-04T04:51:17.628 DEBUG:teuthology.orchestra.console.smithi107:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:51:17.628 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T04:51:32.554 DEBUG:teuthology.provision.fog.smithi064:FOG request: GET /task/active None
2023-11-04T04:51:32.557 DEBUG:teuthology.provision.fog.smithi012:FOG request: GET /task/active None
2023-11-04T04:51:32.634 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:51:35.706 WARNING:teuthology.provision.fog.smithi012:[Errno None] Unable to connect to port 22 on 172.21.15.12
2023-11-04T04:51:37.700 DEBUG:teuthology.orchestra.console.smithi107:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:51:37.700 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T04:51:41.711 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:51:44.766 WARNING:teuthology.provision.fog.smithi012:[Errno None] Unable to connect to port 22 on 172.21.15.12
2023-11-04T04:51:47.640 DEBUG:teuthology.provision.fog.smithi064:FOG request: GET /task/active None
2023-11-04T04:51:47.716 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:51:50.773 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:51:50.778 WARNING:teuthology.provision.fog.smithi064:[Errno None] Unable to connect to port 22 on 172.21.15.64
2023-11-04T04:51:53.850 WARNING:teuthology.provision.fog.smithi012:[Errno None] Unable to connect to port 22 on 172.21.15.12
2023-11-04T04:51:56.782 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:51:57.775 DEBUG:teuthology.orchestra.console.smithi107:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:51:57.775 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:51:58.785 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:51:59.834 WARNING:teuthology.provision.fog.smithi064:[Errno None] Unable to connect to port 22 on 172.21.15.64
2023-11-04T04:51:59.852 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:52:02.789 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:52:02.910 WARNING:teuthology.provision.fog.smithi012:[Errno None] Unable to connect to port 22 on 172.21.15.12
2023-11-04T04:52:03.900 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:52:05.836 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:52:07.902 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:52:08.890 WARNING:teuthology.provision.fog.smithi064:[Errno None] Unable to connect to port 22 on 172.21.15.64
2023-11-04T04:52:08.913 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:52:09.011 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:52:11.994 WARNING:teuthology.provision.fog.smithi012:[Errno None] Unable to connect to port 22 on 172.21.15.12
2023-11-04T04:52:13.013 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:52:14.122 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:52:14.893 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:52:17.950 WARNING:teuthology.provision.fog.smithi064:[Errno None] Unable to connect to port 22 on 172.21.15.64
2023-11-04T04:52:17.995 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:52:18.123 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:52:19.234 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:52:23.238 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:52:23.953 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:52:24.348 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:52:27.038 WARNING:teuthology.provision.fog.smithi064:[Errno None] Unable to connect to port 22 on 172.21.15.64
2023-11-04T04:52:28.350 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:52:29.459 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:52:33.043 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:52:33.461 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:52:34.571 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:52:36.122 WARNING:teuthology.provision.fog.smithi064:[Errno None] Unable to connect to port 22 on 172.21.15.64
2023-11-04T04:52:38.574 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:52:39.682 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:52:42.127 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:52:43.685 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:52:44.793 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:52:48.798 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:52:49.907 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:52:53.908 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:52:55.018 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:52:59.022 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:53:00.130 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:53:04.134 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:53:05.244 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:53:09.246 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:53:10.356 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:53:10.357 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T04:53:10.357 DEBUG:teuthology.provision.fog.smithi107:Canceling deploy task with ID 582607
2023-11-04T04:53:10.357 DEBUG:teuthology.provision.fog.smithi107:FOG request: DELETE /task/cancel {"id": 582607}
2023-11-04T04:53:10.407 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi107
2023-11-04T04:53:10.407 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T04:53:10.561 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3729f464412345708ccaed10afc8e353
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi107
2023-11-04T04:53:10.563 INFO:teuthology.nuke:Checking targets against current locks
2023-11-04T04:53:10.618 DEBUG:teuthology.nuke:shortname: smithi012
2023-11-04T04:53:10.619 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-04T04:53:10.645 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-11-04_04:30:51-fs-rishabh-2023nov3-testing-default-smithi/7447087', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-04 04:48:43.199653', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDcrBil5ugfv6dDyA9EIiGquZOp8xzu5/RCxBZm1o79aW+VsTj7Bs+dizTluJp6DO4jGvAKLTa3MUZWRWz4Jw/4='}
2023-11-04T04:53:10.649 DEBUG:teuthology.nuke:shortname: smithi064
2023-11-04T04:53:10.649 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-04T04:53:10.670 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-11-04_04:30:51-fs-rishabh-2023nov3-testing-default-smithi/7447087', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-04 04:48:43.197853', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN1rPcnr9S2bVDu/V9RvRgp11Qk0ib3+HQ8NGjvrcsJIh1ObvJkPrkAJQnz6HJZS2YHi6xtZZXNKwRx7h6ZR8YA='}
2023-11-04T04:53:10.677 DEBUG:teuthology.nuke:shortname: smithi107
2023-11-04T04:53:10.677 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-04T04:53:10.705 INFO:teuthology.orchestra.console.smithi012:Power off
2023-11-04T04:53:10.706 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:53:10.715 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-11-04_04:30:51-fs-rishabh-2023nov3-testing-default-smithi/7447087', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-04 04:48:43.201086', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLHih7/bY46unkV8HdvRQITsI4kzgIf7ijYt94oJFcW75KCV3HlGe4BlIWExlgPMf6YBjwotlhVuY8v9dxqqHwc='}
2023-11-04T04:53:10.728 DEBUG:teuthology.orchestra.console.smithi012:power off output: Chassis Power Control: Down/Off
2023-11-04T04:53:10.728 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:53:10.743 INFO:teuthology.orchestra.console.smithi064:Power off
2023-11-04T04:53:10.744 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:53:10.756 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on
2023-11-04T04:53:10.766 DEBUG:teuthology.orchestra.console.smithi064:power off output: Chassis Power Control: Down/Off
2023-11-04T04:53:10.766 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:53:10.782 INFO:teuthology.orchestra.console.smithi107:Power off
2023-11-04T04:53:10.783 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:53:10.792 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on
2023-11-04T04:53:14.758 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:53:14.793 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:53:14.868 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on
2023-11-04T04:53:14.904 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on
2023-11-04T04:53:17.998 WARNING:teuthology.provision.fog.smithi012:timed out
2023-11-04T04:53:18.871 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:53:18.904 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:53:18.982 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is off
2023-11-04T04:53:19.015 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is off
2023-11-04T04:53:19.083 INFO:teuthology.orchestra.console.smithi012:Power off completed
2023-11-04T04:53:19.116 INFO:teuthology.orchestra.console.smithi064:Power off completed
2023-11-04T04:53:19.248 INFO:teuthology.lock.ops:unlocked: smithi012.front.sepia.ceph.com
2023-11-04T04:53:19.265 INFO:teuthology.lock.ops:unlocked: smithi064.front.sepia.ceph.com
2023-11-04T04:53:24.003 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:53:30.853 DEBUG:teuthology.orchestra.console.smithi107:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:53:30.854 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:53:42.138 WARNING:teuthology.provision.fog.smithi064:timed out
2023-11-04T04:53:42.394 WARNING:teuthology.provision.fog.smithi012:[Errno None] Unable to connect to port 22 on 172.21.15.12
2023-11-04T04:53:48.147 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:53:48.396 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:53:50.925 DEBUG:teuthology.orchestra.console.smithi107:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:53:50.925 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:53:51.450 WARNING:teuthology.provision.fog.smithi012:[Errno None] Unable to connect to port 22 on 172.21.15.12
2023-11-04T04:53:57.455 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:54:00.506 WARNING:teuthology.provision.fog.smithi012:[Errno None] Unable to connect to port 22 on 172.21.15.12
2023-11-04T04:54:06.514 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:54:06.718 WARNING:teuthology.provision.fog.smithi064:[Errno None] Unable to connect to port 22 on 172.21.15.64
2023-11-04T04:54:09.597 WARNING:teuthology.provision.fog.smithi012:[Errno None] Unable to connect to port 22 on 172.21.15.12
2023-11-04T04:54:10.996 DEBUG:teuthology.orchestra.console.smithi107:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:54:10.997 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:54:12.006 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:54:12.721 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:54:15.599 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:54:15.802 WARNING:teuthology.provision.fog.smithi064:[Errno None] Unable to connect to port 22 on 172.21.15.64
2023-11-04T04:54:16.008 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:54:17.118 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:54:18.654 WARNING:teuthology.provision.fog.smithi012:[Errno None] Unable to connect to port 22 on 172.21.15.12
2023-11-04T04:54:21.121 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:54:21.805 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:54:22.230 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:54:24.658 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:54:24.862 WARNING:teuthology.provision.fog.smithi064:[Errno None] Unable to connect to port 22 on 172.21.15.64
2023-11-04T04:54:26.234 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:54:27.346 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:54:30.867 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:54:31.348 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:54:32.458 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:54:33.950 WARNING:teuthology.provision.fog.smithi064:[Errno None] Unable to connect to port 22 on 172.21.15.64
2023-11-04T04:54:36.462 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:54:37.573 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:54:39.955 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:54:41.575 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:54:42.690 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:54:43.034 WARNING:teuthology.provision.fog.smithi064:[Errno None] Unable to connect to port 22 on 172.21.15.64
2023-11-04T04:54:43.066 WARNING:teuthology.provision.fog.smithi012:[Errno None] Unable to connect to port 22 on 172.21.15.12
2023-11-04T04:54:46.694 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:54:47.806 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:54:49.036 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:54:49.068 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:54:51.810 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:54:52.122 WARNING:teuthology.provision.fog.smithi012:[Errno None] Unable to connect to port 22 on 172.21.15.12
2023-11-04T04:54:52.920 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:54:56.922 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:54:58.033 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:54:58.125 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:54:58.250 ERROR:teuthology.orchestra.connection:Error authenticating with smithi012.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2023-11-04T04:55:02.035 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:55:02.341 ERROR:teuthology.orchestra.connection:Error authenticating with smithi012.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2023-11-04T04:55:03.145 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:55:07.146 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:55:07.386 WARNING:teuthology.provision.fog.smithi064:[Errno None] Unable to connect to port 22 on 172.21.15.64
2023-11-04T04:55:08.256 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:55:09.466 ERROR:teuthology.orchestra.connection:Error authenticating with smithi012.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2023-11-04T04:55:12.257 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:55:13.367 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:55:13.389 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:55:13.394 WARNING:teuthology.provision.fog.smithi064:[Errno None] Unable to connect to port 22 on 172.21.15.64
2023-11-04T04:55:17.369 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:55:18.480 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:55:19.396 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:55:19.525 ERROR:teuthology.orchestra.connection:Error authenticating with smithi064.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2023-11-04T04:55:19.556 ERROR:teuthology.orchestra.connection:Error authenticating with smithi012.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2023-11-04T04:55:22.481 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:55:23.593 DEBUG:teuthology.orchestra.console.smithi107:check power output: 
2023-11-04T04:55:23.594 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T04:55:23.595 ERROR:teuthology.orchestra.console.smithi107:Failed to power off
2023-11-04T04:55:23.621 ERROR:teuthology.orchestra.connection:Error authenticating with smithi064.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2023-11-04T04:55:23.750 INFO:teuthology.lock.ops:unlocked: smithi107.front.sepia.ceph.com
2023-11-04T04:55:24.099 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi107