2023-06-28T07:21:03.414 INFO:root:teuthology version: 0.0.1.dev132+gaa9345a 2023-06-28T07:21:03.416 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-28T07:21:03.416 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-28T07:21:38.152 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd91f33d3a0>, signals=[15]) 2023-06-28T07:21:38.153 INFO:teuthology.lock.ops:Start node 'smithi031.front.sepia.ceph.com' reimaging 2023-06-28T07:21:38.154 INFO:teuthology.lock.ops:Updating [smithi031.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T07:21:38.154 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2023-06-28T07:21:43.031 INFO:teuthology.lock.ops:Start node 'smithi150.front.sepia.ceph.com' reimaging 2023-06-28T07:21:43.031 INFO:teuthology.lock.ops:Updating [smithi150.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T07:21:43.031 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2023-06-28T07:21:43.033 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /host {"name": "smithi031"} 2023-06-28T07:21:53.062 INFO:teuthology.lock.ops:Start node 'smithi156.front.sepia.ceph.com' reimaging 2023-06-28T07:21:53.063 INFO:teuthology.lock.ops:Updating [smithi156.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T07:21:53.063 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server 2023-06-28T07:21:53.065 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /host {"name": "smithi150"} 2023-06-28T07:21:58.214 INFO:teuthology.lock.ops:Start node 'smithi195.front.sepia.ceph.com' reimaging 2023-06-28T07:21:58.215 INFO:teuthology.lock.ops:Updating [smithi195.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T07:21:58.215 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2023-06-28T07:21:58.216 DEBUG:teuthology.provision.fog.smithi156:FOG request: GET /host {"name": "smithi156"} 2023-06-28T07:22:08.232 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /image {"name": "smithi_ubuntu_20.04"} 2023-06-28T07:22:08.234 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /host {"name": "smithi195"} 2023-06-28T07:22:23.254 DEBUG:teuthology.provision.fog.smithi031:Requesting image smithi_ubuntu_20.04 (ID 2) 2023-06-28T07:22:23.254 DEBUG:teuthology.provision.fog.smithi031:FOG request: PUT /host/136 {"imageID": 2} 2023-06-28T07:22:23.256 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /image {"name": "smithi_ubuntu_20.04"} 2023-06-28T07:22:38.274 INFO:teuthology.provision.fog.smithi031:Scheduling deploy of ubuntu 20.04 2023-06-28T07:22:38.275 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None 2023-06-28T07:22:38.276 DEBUG:teuthology.provision.fog.smithi156:FOG request: GET /image {"name": "smithi_ubuntu_20.04"} 2023-06-28T07:22:48.216 DEBUG:teuthology.provision.fog.smithi150:Requesting image smithi_ubuntu_20.04 (ID 2) 2023-06-28T07:22:48.216 DEBUG:teuthology.provision.fog.smithi150:FOG request: PUT /host/255 {"imageID": 2} 2023-06-28T07:22:48.242 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /image {"name": "smithi_ubuntu_20.04"} 2023-06-28T07:22:48.271 DEBUG:teuthology.provision.fog.smithi156:Requesting image smithi_ubuntu_20.04 (ID 2) 2023-06-28T07:22:48.272 DEBUG:teuthology.provision.fog.smithi156:FOG request: PUT /host/261 {"imageID": 2} 2023-06-28T07:22:48.276 INFO:teuthology.provision.fog.smithi150:Scheduling deploy of ubuntu 20.04 2023-06-28T07:22:48.276 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None 2023-06-28T07:22:48.282 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T07:22:48.285 DEBUG:teuthology.provision.fog.smithi195:Requesting image smithi_ubuntu_20.04 (ID 2) 2023-06-28T07:22:48.285 DEBUG:teuthology.provision.fog.smithi195:FOG request: PUT /host/300 {"imageID": 2} 2023-06-28T07:22:48.311 DEBUG:teuthology.provision.fog.smithi031:FOG request: POST /host/136/task {"taskTypeID": 1} 2023-06-28T07:22:48.326 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T07:22:48.327 INFO:teuthology.provision.fog.smithi156:Scheduling deploy of ubuntu 20.04 2023-06-28T07:22:48.327 DEBUG:teuthology.provision.fog.smithi156:FOG request: GET /task/active None 2023-06-28T07:22:48.330 INFO:teuthology.provision.fog.smithi195:Scheduling deploy of ubuntu 20.04 2023-06-28T07:22:48.330 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /task/active None 2023-06-28T07:22:48.368 DEBUG:teuthology.provision.fog.smithi150:FOG request: POST /host/255/task {"taskTypeID": 1} 2023-06-28T07:22:53.374 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None 2023-06-28T07:22:53.376 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T07:22:53.377 DEBUG:teuthology.provision.fog.smithi156:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T07:22:58.427 DEBUG:teuthology.provision.fog.smithi156:FOG request: POST /host/261/task {"taskTypeID": 1} 2023-06-28T07:22:58.441 DEBUG:teuthology.provision.fog.smithi195:FOG request: POST /host/300/task {"taskTypeID": 1} 2023-06-28T07:22:58.443 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None 2023-06-28T07:23:00.999 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /task/active None 2023-06-28T07:23:01.001 DEBUG:teuthology.provision.fog.smithi156:FOG request: GET /task/active None 2023-06-28T07:23:01.034 INFO:teuthology.orchestra.console:Power off smithi150 2023-06-28T07:23:01.035 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T07:23:01.049 INFO:teuthology.orchestra.console:Power off smithi031 2023-06-28T07:23:01.050 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T07:23:01.060 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:23:01.079 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:23:01.118 INFO:teuthology.orchestra.console:Power off smithi156 2023-06-28T07:23:01.119 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T07:23:01.132 INFO:teuthology.orchestra.console:Power off smithi195 2023-06-28T07:23:01.133 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T07:23:01.142 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:23:01.154 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:23:01.192 INFO:teuthology.orchestra.console:Power off for smithi150 completed 2023-06-28T07:23:01.201 INFO:teuthology.orchestra.console:Power off for smithi031 completed 2023-06-28T07:23:01.264 INFO:teuthology.orchestra.console:Power off for smithi156 completed 2023-06-28T07:23:01.276 INFO:teuthology.orchestra.console:Power off for smithi195 completed 2023-06-28T07:23:01.293 INFO:teuthology.orchestra.console:Power on smithi150 2023-06-28T07:23:01.293 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T07:23:01.303 INFO:teuthology.orchestra.console:Power on smithi031 2023-06-28T07:23:01.303 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T07:23:01.313 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:23:01.325 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:23:01.365 INFO:teuthology.orchestra.console:Power on smithi156 2023-06-28T07:23:01.365 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T07:23:01.377 INFO:teuthology.orchestra.console:Power on smithi195 2023-06-28T07:23:01.377 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T07:23:01.387 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:23:01.401 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:23:05.336 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:23:05.347 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:23:05.411 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:23:05.423 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:23:10.447 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:23:10.459 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:23:10.523 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:23:10.534 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:23:15.562 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:23:15.570 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:23:15.633 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:23:15.643 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:23:15.844 INFO:teuthology.orchestra.console:Power on for smithi156 completed 2023-06-28T07:23:15.945 INFO:teuthology.provision.fog.smithi156:Waiting for deploy to finish 2023-06-28T07:23:15.945 DEBUG:teuthology.provision.fog.smithi156:FOG request: GET /task/active None 2023-06-28T07:23:21.054 INFO:teuthology.orchestra.console:Power on for smithi150 completed 2023-06-28T07:23:21.054 INFO:teuthology.orchestra.console:Power on for smithi031 completed 2023-06-28T07:23:21.055 INFO:teuthology.orchestra.console:Power on for smithi195 completed 2023-06-28T07:23:21.155 INFO:teuthology.provision.fog.smithi150:Waiting for deploy to finish 2023-06-28T07:23:21.155 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None 2023-06-28T07:23:21.157 INFO:teuthology.provision.fog.smithi031:Waiting for deploy to finish 2023-06-28T07:23:21.157 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None 2023-06-28T07:23:21.158 INFO:teuthology.provision.fog.smithi195:Waiting for deploy to finish 2023-06-28T07:23:21.158 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /task/active None 2023-06-28T07:23:36.022 DEBUG:teuthology.provision.fog.smithi156:FOG request: GET /task/active None 2023-06-28T07:23:41.033 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:23:56.040 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None 2023-06-28T07:24:01.045 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /task/active None 2023-06-28T07:24:06.049 DEBUG:teuthology.provision.fog.smithi156:FOG request: GET /task/active None 2023-06-28T07:24:20.724 WARNING:teuthology.provision.fog.smithi031:[Errno None] Unable to connect to port 22 on 172.21.15.31 2023-06-28T07:24:26.060 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None 2023-06-28T07:24:31.069 DEBUG:teuthology.provision.fog.smithi156:FOG request: GET /task/active None 2023-06-28T07:24:31.070 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /task/active None 2023-06-28T07:24:31.071 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:24:41.159 ERROR:teuthology.orchestra.connection:Error authenticating with smithi031.front.sepia.ceph.com: Authentication failed. 2023-06-28T07:24:41.160 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-06-28T07:24:41.161 DEBUG:teuthology.orchestra.run.smithi031:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-06-28T07:24:41.164 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:24:42.170 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:24:43.175 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:24:44.180 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:24:45.185 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:24:46.191 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:24:47.197 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:24:48.202 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:24:49.207 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:24:50.212 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:24:51.218 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:24:52.223 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:24:53.228 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:24:54.233 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:24:55.238 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:24:56.165 DEBUG:teuthology.provision.fog.smithi156:FOG request: GET /task/active None 2023-06-28T07:25:01.173 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /task/active None 2023-06-28T07:25:01.174 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None 2023-06-28T07:25:06.748 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:06.849 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:07.853 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:08.857 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:09.861 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:10.865 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:11.869 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:12.874 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:13.879 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:14.883 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:15.888 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:16.893 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:17.898 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:18.903 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:19.908 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:20.913 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:21.821 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None 2023-06-28T07:25:26.829 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /task/active None 2023-06-28T07:25:26.831 DEBUG:teuthology.provision.fog.smithi156:FOG request: GET /task/active None 2023-06-28T07:25:36.841 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:36.943 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:37.948 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:38.953 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:39.958 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:40.963 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:41.968 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:42.973 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:43.978 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:44.982 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:45.988 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:46.993 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:47.998 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:49.004 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:50.009 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:51.014 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:51.912 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None 2023-06-28T07:25:51.915 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /task/active None 2023-06-28T07:25:51.915 DEBUG:teuthology.provision.fog.smithi156:FOG request: GET /task/active None 2023-06-28T07:25:52.020 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:53.025 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:54.031 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:55.036 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:56.042 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:57.047 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:58.052 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:25:59.057 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:26:00.063 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:26:01.068 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:26:02.073 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:26:03.078 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:26:04.083 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:26:05.088 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:26:06.093 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:26:06.988 DEBUG:teuthology.provision.fog.smithi156:FOG request: GET /task/active None 2023-06-28T07:26:11.993 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /task/active None 2023-06-28T07:26:11.994 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None 2023-06-28T07:26:20.237 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:26:20.338 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:26:21.342 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:26:22.346 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:26:23.350 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:26:24.356 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:26:25.361 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:26:26.366 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:26:27.371 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:26:28.376 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:26:35.315 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /task/active None 2023-06-28T07:26:40.323 DEBUG:teuthology.provision.fog.smithi156:FOG request: GET /task/active None 2023-06-28T07:26:40.325 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None 2023-06-28T07:26:56.420 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /task/active None 2023-06-28T07:26:56.422 DEBUG:teuthology.provision.fog.smithi156:FOG request: GET /task/active None 2023-06-28T07:26:56.427 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None 2023-06-28T07:26:56.498 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:26:56.504 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:26:56.508 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:27:15.056 WARNING:teuthology.provision.fog.smithi195:[Errno None] Unable to connect to port 22 on 172.21.15.195 2023-06-28T07:27:15.057 WARNING:teuthology.provision.fog.smithi156:[Errno None] Unable to connect to port 22 on 172.21.15.156 2023-06-28T07:27:15.057 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-06-28T07:27:21.058 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:27:21.060 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:27:21.062 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:27:24.116 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-06-28T07:27:24.117 WARNING:teuthology.provision.fog.smithi195:[Errno None] Unable to connect to port 22 on 172.21.15.195 2023-06-28T07:27:24.117 WARNING:teuthology.provision.fog.smithi156:[Errno None] Unable to connect to port 22 on 172.21.15.156 2023-06-28T07:27:30.118 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:27:30.120 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:27:30.122 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:27:33.168 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-06-28T07:27:33.200 WARNING:teuthology.provision.fog.smithi195:[Errno None] Unable to connect to port 22 on 172.21.15.195 2023-06-28T07:27:33.201 WARNING:teuthology.provision.fog.smithi156:[Errno None] Unable to connect to port 22 on 172.21.15.156 2023-06-28T07:27:39.169 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:27:44.176 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:27:49.181 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:27:49.183 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-06-28T07:27:52.240 WARNING:teuthology.provision.fog.smithi156:[Errno None] Unable to connect to port 22 on 172.21.15.156 2023-06-28T07:27:55.185 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:28:00.191 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:28:07.536 WARNING:teuthology.provision.fog.smithi195:[Errno None] Unable to connect to port 22 on 172.21.15.195 2023-06-28T07:28:13.538 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:28:18.541 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2023-06-28T07:28:18.544 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 295, in _wait_for_ready self.remote.run(args=cmd, timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 97, in execute self.client.exec_command(self.command, timeout=self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 510, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 919, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1050, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2109, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) ConnectionResetError: [Errno 104] Connection reset by peer 2023-06-28T07:28:18.545 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-28T07:28:18.546 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-28T07:28:23.553 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-06-28T07:28:23.635 INFO:teuthology.nuke:Checking targets against current locks 2023-06-28T07:28:28.641 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-06-28T07:28:28.642 DEBUG:teuthology.orchestra.run.smithi195:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-06-28T07:28:32.749 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:28:32.780 DEBUG:teuthology.nuke:shortname: smithi031 2023-06-28T07:28:32.781 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T07:28:32.803 DEBUG:teuthology.nuke:shortname: smithi150 2023-06-28T07:28:32.803 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T07:28:32.811 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-27_22:53:17-rbd-reef-release-distro-default-smithi/7319114', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 07:21:02.578708', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0='} 2023-06-28T07:28:32.828 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-27_22:53:17-rbd-reef-release-distro-default-smithi/7319114', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 07:21:02.580526', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-06-28T07:28:32.835 DEBUG:teuthology.nuke:shortname: smithi156 2023-06-28T07:28:32.835 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T07:28:32.851 DEBUG:teuthology.nuke:shortname: smithi195 2023-06-28T07:28:32.852 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T07:28:32.863 INFO:teuthology.orchestra.console:Power off smithi031 2023-06-28T07:28:32.864 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T07:28:32.880 INFO:teuthology.orchestra.console:Power off smithi150 2023-06-28T07:28:32.880 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T07:28:32.888 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-27_22:53:17-rbd-reef-release-distro-default-smithi/7319114', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 07:21:02.582093', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-06-28T07:28:32.891 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-27_22:53:17-rbd-reef-release-distro-default-smithi/7319114', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 07:21:02.576935', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTmLPrv5YXCUEcH3IArOW9fhnaAouznAnWaIH6xGTXhaSHQAiMZBbbrklW6pQYRFfbU9BwDJwyVse/Nd3FflMxGttUKlM1KjRc5F+B/byny3KOk2VGL/ODyBFIIQgUAapRhsxEjOt7Jk2j6b3P/WKI0e6CiZfpgOry9Nqic/YLw/zpoqFSWHl0BIqlfitIKyPyyB6Kx5jN9yyC1hfI+3uwJ9pVsZFqdcC6YRzDGUVM5dzeum62rbZ/gO7tPzFtwmiHlcSNCTXd72AEv8ndY2+u9kyxHmgZM5rEaYTTTCrC4a7bST3rR3dMYC3rIrdwnjN2RxYEVBirUD2v/LZ1bZPzsW2sfYylMDKiVUm9r1LNZelZWuWb3E/74CJ89Y3Utt2m+hIXYO8VhbT9z3BYxcoJgsaLWsBbrpFCKNfFu1wvSEbPOfqF9rmYSzhhUrVaH/2uSjcB1lgcEtHDoSMsiOR7G6ggp2xmFhnFEaTKTahvmwQOmHC3qfCu9ZPYyIO2460='} 2023-06-28T07:28:32.894 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:28:32.904 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:28:32.951 INFO:teuthology.orchestra.console:Power off smithi156 2023-06-28T07:28:32.952 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T07:28:32.969 INFO:teuthology.orchestra.console:Power off smithi195 2023-06-28T07:28:32.969 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T07:28:32.979 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:28:32.993 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:28:33.098 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-06-28T07:28:33.099 DEBUG:teuthology.orchestra.run.smithi150:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-06-28T07:28:36.469 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-28T07:28:36.625 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-28T07:28:36.720 WARNING:teuthology.provision.fog.smithi156:[Errno None] Unable to connect to port 22 on 172.21.15.156 2023-06-28T07:28:36.919 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:28:36.929 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:28:37.004 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:28:37.018 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:28:42.032 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:28:42.041 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:28:42.115 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:28:42.129 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:28:42.341 INFO:teuthology.orchestra.console:Power off for smithi195 completed 2023-06-28T07:28:45.092 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:28:45.138 INFO:teuthology.lock.ops:unlocked: smithi195.front.sepia.ceph.com 2023-06-28T07:28:49.096 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:28:49.105 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:28:49.113 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:28:54.208 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:28:54.217 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:28:54.227 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:28:59.321 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:28:59.329 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:28:59.338 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:29:03.600 WARNING:teuthology.provision.fog.smithi156:[Errno None] Unable to connect to port 22 on 172.21.15.156 2023-06-28T07:29:04.433 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:29:04.441 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:29:04.450 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:29:04.644 INFO:teuthology.orchestra.console:Power off for smithi031 completed 2023-06-28T07:29:04.653 INFO:teuthology.orchestra.console:Power off for smithi150 completed 2023-06-28T07:29:09.755 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:29:19.768 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:29:19.775 WARNING:teuthology.provision.fog.smithi156:[Errno None] Unable to connect to port 22 on 172.21.15.156 2023-06-28T07:29:22.943 INFO:teuthology.lock.ops:unlocked: smithi150.front.sepia.ceph.com 2023-06-28T07:29:22.944 INFO:teuthology.lock.ops:unlocked: smithi031.front.sepia.ceph.com 2023-06-28T07:29:23.010 INFO:teuthology.orchestra.console:Power off for smithi156 completed 2023-06-28T07:29:28.121 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:29:33.153 WARNING:teuthology.provision.fog.smithi156:[Errno None] Unable to connect to port 22 on 172.21.15.156 2023-06-28T07:29:33.182 INFO:teuthology.lock.ops:unlocked: smithi156.front.sepia.ceph.com 2023-06-28T07:29:38.680 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 57, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 295, in _wait_for_ready self.remote.run(args=cmd, timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 97, in execute self.client.exec_command(self.command, timeout=self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 510, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 919, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1050, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2109, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) ConnectionResetError: [Errno 104] Connection reset by peer