2023-08-08T01:07:53.742 INFO:root:teuthology version: 0.0.1.dev181+g7fda959
2023-08-08T01:07:53.744 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-08-08T01:07:53.937 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7fc94a924850>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7fc94bbd2280>, signals=[15])
2023-08-08T01:07:53.939 INFO:teuthology.lock.ops:Start node 'smithi105.front.sepia.ceph.com' reimaging
2023-08-08T01:07:53.939 INFO:teuthology.lock.ops:Updating [smithi105.front.sepia.ceph.com]: reset os type and version on server
2023-08-08T01:07:53.939 INFO:teuthology.lock.ops:Updating smithi105.front.sepia.ceph.com on lock server
2023-08-08T01:07:53.960 INFO:teuthology.lock.ops:Start node 'smithi140.front.sepia.ceph.com' reimaging
2023-08-08T01:07:53.960 INFO:teuthology.lock.ops:Updating [smithi140.front.sepia.ceph.com]: reset os type and version on server
2023-08-08T01:07:53.961 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server
2023-08-08T01:07:53.962 DEBUG:teuthology.provision.fog.smithi105:FOG request: GET /host {"name": "smithi105"}
2023-08-08T01:07:53.984 INFO:teuthology.lock.ops:Start node 'smithi203.front.sepia.ceph.com' reimaging
2023-08-08T01:07:53.985 INFO:teuthology.lock.ops:Updating [smithi203.front.sepia.ceph.com]: reset os type and version on server
2023-08-08T01:07:53.985 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server
2023-08-08T01:07:53.986 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /host {"name": "smithi140"}
2023-08-08T01:07:54.010 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /host {"name": "smithi203"}
2023-08-08T01:07:54.035 DEBUG:teuthology.provision.fog.smithi105:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-08-08T01:07:54.048 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-08-08T01:07:54.071 DEBUG:teuthology.provision.fog.smithi105:Requesting image smithi_rhel_8.6 (ID 5)
2023-08-08T01:07:54.072 DEBUG:teuthology.provision.fog.smithi105:FOG request: PUT /host/210 {"imageID": 5}
2023-08-08T01:07:54.077 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-08-08T01:07:54.088 DEBUG:teuthology.provision.fog.smithi140:Requesting image smithi_rhel_8.6 (ID 5)
2023-08-08T01:07:54.089 DEBUG:teuthology.provision.fog.smithi140:FOG request: PUT /host/245 {"imageID": 5}
2023-08-08T01:07:54.114 DEBUG:teuthology.provision.fog.smithi203:Requesting image smithi_rhel_8.6 (ID 5)
2023-08-08T01:07:54.115 DEBUG:teuthology.provision.fog.smithi203:FOG request: PUT /host/307 {"imageID": 5}
2023-08-08T01:07:54.121 INFO:teuthology.provision.fog.smithi105:Scheduling deploy of rhel 8.6
2023-08-08T01:07:54.122 DEBUG:teuthology.provision.fog.smithi105:FOG request: GET /task/active None
2023-08-08T01:07:54.133 INFO:teuthology.provision.fog.smithi140:Scheduling deploy of rhel 8.6
2023-08-08T01:07:54.134 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-08T01:07:54.158 INFO:teuthology.provision.fog.smithi203:Scheduling deploy of rhel 8.6
2023-08-08T01:07:54.159 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-08-08T01:07:54.162 DEBUG:teuthology.provision.fog.smithi105:FOG request: GET /tasktype {"name": "deploy"}
2023-08-08T01:07:54.176 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /tasktype {"name": "deploy"}
2023-08-08T01:07:54.200 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /tasktype {"name": "deploy"}
2023-08-08T01:07:54.207 DEBUG:teuthology.provision.fog.smithi105:FOG request: POST /host/210/task {"taskTypeID": 1}
2023-08-08T01:07:54.218 DEBUG:teuthology.provision.fog.smithi140:FOG request: POST /host/245/task {"taskTypeID": 1}
2023-08-08T01:07:54.235 DEBUG:teuthology.provision.fog.smithi203:FOG request: POST /host/307/task {"taskTypeID": 1}
2023-08-08T01:07:54.252 DEBUG:teuthology.provision.fog.smithi105:FOG request: GET /task/active None
2023-08-08T01:07:54.289 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-08T01:07:54.309 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-08-08T01:07:54.326 INFO:teuthology.orchestra.console.smithi105:Power off
2023-08-08T01:07:54.327 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-08-08T01:07:54.347 DEBUG:teuthology.orchestra.console.smithi105:power off output: Chassis Power Control: Down/Off
2023-08-08T01:07:54.347 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-08T01:07:54.367 INFO:teuthology.orchestra.console.smithi140:Power off
2023-08-08T01:07:54.368 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-08-08T01:07:54.376 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is off
2023-08-08T01:07:54.388 DEBUG:teuthology.orchestra.console.smithi140:power off output: Chassis Power Control: Down/Off
2023-08-08T01:07:54.388 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-08T01:07:54.404 INFO:teuthology.orchestra.console.smithi203:Power off
2023-08-08T01:07:54.405 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-08-08T01:07:54.415 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is off
2023-08-08T01:07:54.427 DEBUG:teuthology.orchestra.console.smithi203:power off output: Chassis Power Control: Down/Off
2023-08-08T01:07:54.427 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-08T01:07:54.447 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on
2023-08-08T01:07:54.478 INFO:teuthology.orchestra.console.smithi105:Power off completed
2023-08-08T01:07:54.516 INFO:teuthology.orchestra.console.smithi140:Power off completed
2023-08-08T01:07:54.579 INFO:teuthology.orchestra.console.smithi105:Power on
2023-08-08T01:07:54.580 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-08-08T01:07:54.601 DEBUG:teuthology.orchestra.console.smithi105:power on output: Chassis Power Control: Up/On
2023-08-08T01:07:54.602 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-08T01:07:54.618 INFO:teuthology.orchestra.console.smithi140:Power on
2023-08-08T01:07:54.619 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-08-08T01:07:54.631 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is off
2023-08-08T01:07:54.643 DEBUG:teuthology.orchestra.console.smithi140:power on output: Chassis Power Control: Up/On
2023-08-08T01:07:54.644 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-08T01:07:54.665 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is off
2023-08-08T01:07:58.452 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-08T01:07:58.562 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on
2023-08-08T01:07:58.632 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-08T01:07:58.667 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-08T01:07:58.742 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is off
2023-08-08T01:07:58.777 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is on
2023-08-08T01:07:58.878 INFO:teuthology.orchestra.console.smithi140:Power on completed
2023-08-08T01:07:58.979 INFO:teuthology.provision.fog.smithi140:Waiting for deploy to finish
2023-08-08T01:07:58.979 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-08T01:08:02.567 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-08T01:08:02.676 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on
2023-08-08T01:08:02.744 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-08T01:08:02.856 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on
2023-08-08T01:08:02.957 INFO:teuthology.orchestra.console.smithi105:Power on completed
2023-08-08T01:08:03.058 INFO:teuthology.provision.fog.smithi105:Waiting for deploy to finish
2023-08-08T01:08:03.059 DEBUG:teuthology.provision.fog.smithi105:FOG request: GET /task/active None
2023-08-08T01:08:06.680 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-08T01:08:06.791 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off
2023-08-08T01:08:06.892 INFO:teuthology.orchestra.console.smithi203:Power off completed
2023-08-08T01:08:06.993 INFO:teuthology.orchestra.console.smithi203:Power on
2023-08-08T01:08:06.993 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-08-08T01:08:07.014 DEBUG:teuthology.orchestra.console.smithi203:power on output: Chassis Power Control: Up/On
2023-08-08T01:08:07.015 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-08T01:08:07.036 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off
2023-08-08T01:08:11.040 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-08T01:08:11.151 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on
2023-08-08T01:08:11.252 INFO:teuthology.orchestra.console.smithi203:Power on completed
2023-08-08T01:08:11.352 INFO:teuthology.provision.fog.smithi203:Waiting for deploy to finish
2023-08-08T01:08:11.353 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-08-08T01:08:14.040 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-08T01:08:18.124 DEBUG:teuthology.provision.fog.smithi105:FOG request: GET /task/active None
2023-08-08T01:08:26.424 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-08-08T01:08:29.106 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-08T01:08:33.193 DEBUG:teuthology.provision.fog.smithi105:FOG request: GET /task/active None
2023-08-08T01:08:41.494 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-08-08T01:08:44.171 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-08T01:08:48.262 DEBUG:teuthology.provision.fog.smithi105:FOG request: GET /task/active None
2023-08-08T01:08:56.564 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-08-08T01:08:59.235 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-08T01:09:03.324 DEBUG:teuthology.provision.fog.smithi105:FOG request: GET /task/active None
2023-08-08T01:09:11.632 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-08-08T01:09:14.296 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-08T01:09:18.388 DEBUG:teuthology.provision.fog.smithi105:FOG request: GET /task/active None
2023-08-08T01:09:26.716 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-08-08T01:09:29.360 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-08T01:09:33.452 DEBUG:teuthology.provision.fog.smithi105:FOG request: GET /task/active None
2023-08-08T01:09:41.780 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-08-08T01:09:44.445 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-08T01:09:48.530 DEBUG:teuthology.provision.fog.smithi105:FOG request: GET /task/active None
2023-08-08T01:09:56.848 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-08-08T01:09:59.519 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-08T01:10:03.612 DEBUG:teuthology.provision.fog.smithi105:FOG request: GET /task/active None
2023-08-08T01:10:11.920 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-08-08T01:10:14.600 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-08T01:10:18.680 DEBUG:teuthology.provision.fog.smithi105:FOG request: GET /task/active None
2023-08-08T01:10:27.000 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-08-08T01:10:29.668 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-08T01:10:33.760 DEBUG:teuthology.provision.fog.smithi105:FOG request: GET /task/active None
2023-08-08T01:10:42.064 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-08-08T01:10:44.735 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-08T01:10:44.790 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-08T01:10:44.866 ERROR:teuthology.orchestra.connection:Error authenticating with smithi140.front.sepia.ceph.com: Authentication failed.
2023-08-08T01:10:44.868 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-08-08T01:10:44.869 DEBUG:teuthology.orchestra.run.smithi140:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-08-08T01:10:44.870 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-08-08T01:10:48.826 DEBUG:teuthology.provision.fog.smithi105:FOG request: GET /task/active None
2023-08-08T01:10:48.895 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-08T01:10:51.952 WARNING:teuthology.provision.fog.smithi105:[Errno None] Unable to connect to port 22 on 172.21.15.105
2023-08-08T01:10:57.128 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-08-08T01:10:57.954 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-08T01:11:01.008 WARNING:teuthology.provision.fog.smithi105:[Errno None] Unable to connect to port 22 on 172.21.15.105
2023-08-08T01:11:07.009 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-08T01:11:10.064 WARNING:teuthology.provision.fog.smithi105:[Errno None] Unable to connect to port 22 on 172.21.15.105
2023-08-08T01:11:12.177 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-08-08T01:11:12.229 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-08T01:11:15.280 WARNING:teuthology.provision.fog.smithi203:[Errno None] Unable to connect to port 22 on 172.21.15.203
2023-08-08T01:11:16.066 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-08T01:11:19.120 WARNING:teuthology.provision.fog.smithi105:[Errno None] Unable to connect to port 22 on 172.21.15.105
2023-08-08T01:11:21.282 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-08T01:11:24.336 WARNING:teuthology.provision.fog.smithi203:[Errno None] Unable to connect to port 22 on 172.21.15.203
2023-08-08T01:11:25.121 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-08T01:11:28.176 WARNING:teuthology.provision.fog.smithi105:[Errno None] Unable to connect to port 22 on 172.21.15.105
2023-08-08T01:11:30.337 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-08T01:11:33.392 WARNING:teuthology.provision.fog.smithi203:[Errno None] Unable to connect to port 22 on 172.21.15.203
2023-08-08T01:11:34.178 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-08T01:11:37.236 WARNING:teuthology.provision.fog.smithi105:[Errno None] Unable to connect to port 22 on 172.21.15.105
2023-08-08T01:11:39.393 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-08T01:11:42.448 WARNING:teuthology.provision.fog.smithi203:[Errno None] Unable to connect to port 22 on 172.21.15.203
2023-08-08T01:11:43.238 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-08T01:11:48.449 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-08T01:11:51.504 WARNING:teuthology.provision.fog.smithi203:[Errno None] Unable to connect to port 22 on 172.21.15.203
2023-08-08T01:11:57.506 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-08T01:12:00.560 WARNING:teuthology.provision.fog.smithi203:[Errno None] Unable to connect to port 22 on 172.21.15.203
2023-08-08T01:12:06.561 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-08T01:12:09.616 WARNING:teuthology.provision.fog.smithi203:[Errno None] Unable to connect to port 22 on 172.21.15.203
2023-08-08T01:12:15.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-08T01:12:43.239 WARNING:teuthology.provision.fog.smithi105:timed out
2023-08-08T01:12:49.241 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-08T01:12:49.568 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-08-08T01:12:49.568 DEBUG:teuthology.orchestra.run.smithi105:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-08-08T01:12:49.831 INFO:teuthology.provision.fog.smithi105:Node is ready
2023-08-08T01:12:49.832 DEBUG:teuthology.orchestra.run.smithi105:> hostname
2023-08-08T01:12:49.850 INFO:teuthology.orchestra.run.smithi105.stdout:smithi105
2023-08-08T01:12:49.850 DEBUG:teuthology.orchestra.run.smithi105:> grep smithi105 /etc/hosts
2023-08-08T01:12:49.908 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.105 smithi105.front.sepia.ceph.com smithi105
2023-08-08T01:12:49.908 DEBUG:teuthology.orchestra.run.smithi105:> sudo hostname smithi105
2023-08-08T01:12:50.018 DEBUG:teuthology.orchestra.run.smithi105:> sudo sed -i -e 's/smithi105/smithi105/g' /etc/hosts
2023-08-08T01:12:50.046 DEBUG:teuthology.orchestra.run.smithi105:> sudo sed -i -e 's/172.21.15.105/172.21.15.105/g' /etc/hosts
2023-08-08T01:12:50.113 DEBUG:teuthology.orchestra.run.smithi105:> sudo sed -i -e 's/smithi105/smithi105/g' /etc/hostname
2023-08-08T01:12:50.180 DEBUG:teuthology.orchestra.run.smithi105:> sudo hostname smithi105
2023-08-08T01:12:50.245 DEBUG:teuthology.orchestra.run.smithi105:> cat /etc/os-release
2023-08-08T01:12:50.302 INFO:teuthology.orchestra.run.smithi105.stdout:NAME="Red Hat Enterprise Linux"
2023-08-08T01:12:50.302 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION="8.6 (Ootpa)"
2023-08-08T01:12:50.303 INFO:teuthology.orchestra.run.smithi105.stdout:ID="rhel"
2023-08-08T01:12:50.303 INFO:teuthology.orchestra.run.smithi105.stdout:ID_LIKE="fedora"
2023-08-08T01:12:50.303 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION_ID="8.6"
2023-08-08T01:12:50.303 INFO:teuthology.orchestra.run.smithi105.stdout:PLATFORM_ID="platform:el8"
2023-08-08T01:12:50.303 INFO:teuthology.orchestra.run.smithi105.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-08-08T01:12:50.303 INFO:teuthology.orchestra.run.smithi105.stdout:ANSI_COLOR="0;31"
2023-08-08T01:12:50.304 INFO:teuthology.orchestra.run.smithi105.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-08-08T01:12:50.304 INFO:teuthology.orchestra.run.smithi105.stdout:HOME_URL="https://www.redhat.com/"
2023-08-08T01:12:50.304 INFO:teuthology.orchestra.run.smithi105.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-08-08T01:12:50.304 INFO:teuthology.orchestra.run.smithi105.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-08-08T01:12:50.304 INFO:teuthology.orchestra.run.smithi105.stdout:
2023-08-08T01:12:50.304 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-08-08T01:12:50.304 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-08-08T01:12:50.305 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-08-08T01:12:50.305 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-08-08T01:12:50.305 INFO:teuthology.provision.fog.smithi105:Deploy complete!
2023-08-08T01:12:50.307 DEBUG:teuthology.parallel:result is None
2023-08-08T01:13:15.619 WARNING:teuthology.provision.fog.smithi203:timed out
2023-08-08T01:13:21.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-08T01:13:21.943 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-08-08T01:13:21.943 DEBUG:teuthology.orchestra.run.smithi203:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-08-08T01:13:22.227 INFO:teuthology.provision.fog.smithi203:Node is ready
2023-08-08T01:13:22.227 DEBUG:teuthology.orchestra.run.smithi203:> hostname
2023-08-08T01:13:22.245 INFO:teuthology.orchestra.run.smithi203.stdout:smithi203
2023-08-08T01:13:22.245 DEBUG:teuthology.orchestra.run.smithi203:> grep smithi203 /etc/hosts
2023-08-08T01:13:22.303 INFO:teuthology.orchestra.run.smithi203.stdout:172.21.15.203 smithi203.front.sepia.ceph.com smithi203
2023-08-08T01:13:22.303 DEBUG:teuthology.orchestra.run.smithi203:> sudo hostname smithi203
2023-08-08T01:13:22.415 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i -e 's/smithi203/smithi203/g' /etc/hosts
2023-08-08T01:13:22.443 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i -e 's/172.21.15.203/172.21.15.203/g' /etc/hosts
2023-08-08T01:13:22.510 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i -e 's/smithi203/smithi203/g' /etc/hostname
2023-08-08T01:13:22.578 DEBUG:teuthology.orchestra.run.smithi203:> sudo hostname smithi203
2023-08-08T01:13:22.645 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release
2023-08-08T01:13:22.701 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="Red Hat Enterprise Linux"
2023-08-08T01:13:22.701 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="8.6 (Ootpa)"
2023-08-08T01:13:22.701 INFO:teuthology.orchestra.run.smithi203.stdout:ID="rhel"
2023-08-08T01:13:22.701 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE="fedora"
2023-08-08T01:13:22.702 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="8.6"
2023-08-08T01:13:22.702 INFO:teuthology.orchestra.run.smithi203.stdout:PLATFORM_ID="platform:el8"
2023-08-08T01:13:22.702 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-08-08T01:13:22.702 INFO:teuthology.orchestra.run.smithi203.stdout:ANSI_COLOR="0;31"
2023-08-08T01:13:22.702 INFO:teuthology.orchestra.run.smithi203.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-08-08T01:13:22.702 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://www.redhat.com/"
2023-08-08T01:13:22.702 INFO:teuthology.orchestra.run.smithi203.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-08-08T01:13:22.703 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-08-08T01:13:22.703 INFO:teuthology.orchestra.run.smithi203.stdout:
2023-08-08T01:13:22.703 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-08-08T01:13:22.703 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-08-08T01:13:22.703 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-08-08T01:13:22.703 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-08-08T01:13:22.703 INFO:teuthology.provision.fog.smithi203:Deploy complete!
2023-08-08T01:13:22.704 DEBUG:teuthology.parallel:result is None
2023-08-08T01:20:44.911 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/provision/fog.py", line 90, in create
    self._wait_for_ready()
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/provision/fog.py", line 299, in _wait_for_ready
    self.remote.run(args=cmd, timeout=600)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/orchestra/remote.py", line 522, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/orchestra/run.py", line 450, in run
    r.execute()
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/orchestra/run.py", line 97, in execute
    self.client.exec_command(self.command, timeout=self.timeout)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 510, in exec_command
    chan = self._transport.open_session(timeout=timeout)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 919, in open_session
    return self.open_channel(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1054, in open_channel
    raise SSHException("Timeout opening channel.")
paramiko.ssh_exception.SSHException: Timeout opening channel.
2023-08-08T01:20:44.933 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-08-08T01:20:45.219 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=dc492d5d9439442994c5cdac78e6efec
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/provision/fog.py", line 90, in create
    self._wait_for_ready()
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/provision/fog.py", line 299, in _wait_for_ready
    self.remote.run(args=cmd, timeout=600)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/orchestra/remote.py", line 522, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/orchestra/run.py", line 450, in run
    r.execute()
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/orchestra/run.py", line 97, in execute
    self.client.exec_command(self.command, timeout=self.timeout)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 510, in exec_command
    chan = self._transport.open_session(timeout=timeout)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 919, in open_session
    return self.open_channel(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1054, in open_channel
    raise SSHException("Timeout opening channel.")
paramiko.ssh_exception.SSHException: Timeout opening channel.
2023-08-08T01:20:45.223 INFO:teuthology.nuke:Checking targets against current locks
2023-08-08T01:20:45.290 DEBUG:teuthology.nuke:shortname: smithi105
2023-08-08T01:20:45.290 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-08-08T01:20:45.315 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-08-04_18:23:51-fs-wip-rishabh-2023aug3-testing-default-smithi/7361181', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-08 01:07:52.846544', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD+oXyatUS6QUKafdNiXT64Xbm39brcAW6JVTtdPpQVjWFGMJdN8T3Amoudgn2MCUFp1bQg4adC0FGnqKPQPlTw='}
2023-08-08T01:20:45.321 DEBUG:teuthology.nuke:shortname: smithi140
2023-08-08T01:20:45.322 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-08-08T01:20:45.344 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi140.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-08-04_18:23:51-fs-wip-rishabh-2023aug3-testing-default-smithi/7361181', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-08 01:07:52.844131', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOvA4PMkz1bGT4qZCbyXFrr7qrJAa8IgmEXuV8b2a/B6CtSNppxeiInDewzpQj+aWXRluNVhO0GaCsvNbotr2YE='}
2023-08-08T01:20:45.348 DEBUG:teuthology.nuke:shortname: smithi203
2023-08-08T01:20:45.348 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-08-08T01:20:45.380 INFO:teuthology.orchestra.console.smithi105:Power off
2023-08-08T01:20:45.381 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-08-08T01:20:45.392 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-08-04_18:23:51-fs-wip-rishabh-2023aug3-testing-default-smithi/7361181', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-08 01:07:52.845563', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCUcWgX4daBZpgUQLPS8jYbxRooYhXFhWbi6nknpoYAsr+dEJZGO5Fwl8Tg9W12/lVYeDgw9ABpGrJk8LrSCdagSPfJ3j/3uEgGiILxAZrJaCjN0pvjcXktI8SWQjhciu7bCFIZim2e6nLcH7NQqNf+YgbulWbJ0mkk0W4oYW50HKu7LY6JfHNw/1TRLFL562hvDEzkALrWuaS4mD8WAC+/bzM54kotiErC/b3Mx0I6y8nOCrKLB8VE7NGXNfFe4aIMyHS/tm6TMidRDayFh3Jw8GV5QgtmN1V8samAYkCpXl6/ev56MYtWGPILPovsBqS48qcbi5jf085Wc25CNzE8W6/tgJGRHmaI0fUPudWjfr2xgRjgBMaQABz+pR3R0gBUgA52VATUfslgW8NrnNqZNAftKeIwv7NTkAuIts0zvu6Tgqt6fEkisA2u3MlZKmTjNbhfnYoMlaOMSQbdL/sEC0eFwFS4sq8IWT7Gm1xxPEnhE8j1BXTs2YZNSz3o9+c='}
2023-08-08T01:20:45.408 DEBUG:teuthology.orchestra.console.smithi105:power off output: Chassis Power Control: Down/Off
2023-08-08T01:20:45.409 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-08T01:20:45.433 INFO:teuthology.orchestra.console.smithi140:Power off
2023-08-08T01:20:45.434 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-08-08T01:20:45.450 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on
2023-08-08T01:20:45.462 DEBUG:teuthology.orchestra.console.smithi140:power off output: Chassis Power Control: Down/Off
2023-08-08T01:20:45.463 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-08T01:20:45.483 INFO:teuthology.orchestra.console.smithi203:Power off
2023-08-08T01:20:45.484 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-08-08T01:20:45.496 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is on
2023-08-08T01:20:45.512 DEBUG:teuthology.orchestra.console.smithi203:power off output: Chassis Power Control: Down/Off
2023-08-08T01:20:45.512 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-08T01:20:45.540 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on
2023-08-08T01:20:49.451 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-08T01:20:49.498 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-08T01:20:49.542 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-08T01:20:49.568 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on
2023-08-08T01:20:49.610 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is on
2023-08-08T01:20:49.654 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on
2023-08-08T01:20:53.569 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-08T01:20:53.612 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-08T01:20:53.656 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-08T01:20:53.681 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is off
2023-08-08T01:20:53.724 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is off
2023-08-08T01:20:53.769 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on
2023-08-08T01:20:53.782 INFO:teuthology.orchestra.console.smithi105:Power off completed
2023-08-08T01:20:53.826 INFO:teuthology.orchestra.console.smithi140:Power off completed
2023-08-08T01:20:53.940 INFO:teuthology.lock.ops:unlocked: smithi105.front.sepia.ceph.com
2023-08-08T01:20:53.972 INFO:teuthology.lock.ops:unlocked: smithi140.front.sepia.ceph.com
2023-08-08T01:20:57.770 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-08T01:20:57.882 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off
2023-08-08T01:20:57.984 INFO:teuthology.orchestra.console.smithi203:Power off completed
2023-08-08T01:20:58.137 INFO:teuthology.lock.ops:unlocked: smithi203.front.sepia.ceph.com
2023-08-08T01:20:58.480 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/provision/fog.py", line 90, in create
    self._wait_for_ready()
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/provision/fog.py", line 299, in _wait_for_ready
    self.remote.run(args=cmd, timeout=600)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/orchestra/remote.py", line 522, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/orchestra/run.py", line 450, in run
    r.execute()
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/orchestra/run.py", line 97, in execute
    self.client.exec_command(self.command, timeout=self.timeout)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 510, in exec_command
    chan = self._transport.open_session(timeout=timeout)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 919, in open_session
    return self.open_channel(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1054, in open_channel
    raise SSHException("Timeout opening channel.")
paramiko.ssh_exception.SSHException: Timeout opening channel.