2023-06-28T01:41:41.116 INFO:root:teuthology version: 0.0.1.dev130+g3fdf1f2 2023-06-28T01:41:41.118 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-28T01:41:41.118 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-28T01:41:41.330 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f46139544c0>, signals=[15]) 2023-06-28T01:41:41.331 INFO:teuthology.lock.ops:Start node 'smithi050.front.sepia.ceph.com' reimaging 2023-06-28T01:41:41.332 INFO:teuthology.lock.ops:Updating [smithi050.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T01:41:41.332 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server 2023-06-28T01:41:41.354 INFO:teuthology.lock.ops:Start node 'smithi053.front.sepia.ceph.com' reimaging 2023-06-28T01:41:41.354 INFO:teuthology.lock.ops:Updating [smithi053.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T01:41:41.355 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2023-06-28T01:41:41.357 DEBUG:teuthology.provision.fog.smithi050:FOG request: GET /host {"name": "smithi050"} 2023-06-28T01:41:46.393 INFO:teuthology.lock.ops:Start node 'smithi083.front.sepia.ceph.com' reimaging 2023-06-28T01:41:46.394 INFO:teuthology.lock.ops:Updating [smithi083.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T01:41:46.394 INFO:teuthology.lock.ops:Updating smithi083.front.sepia.ceph.com on lock server 2023-06-28T01:41:46.396 DEBUG:teuthology.provision.fog.smithi053:FOG request: GET /host {"name": "smithi053"} 2023-06-28T01:41:51.429 DEBUG:teuthology.provision.fog.smithi050:FOG request: GET /image {"name": "smithi_ubuntu_22.04"} 2023-06-28T01:41:56.865 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /host {"name": "smithi083"} 2023-06-28T01:42:06.905 DEBUG:teuthology.provision.fog.smithi053:FOG request: GET /image {"name": "smithi_ubuntu_22.04"} 2023-06-28T01:42:11.916 DEBUG:teuthology.provision.fog.smithi050:Requesting image smithi_ubuntu_22.04 (ID 6) 2023-06-28T01:42:11.916 DEBUG:teuthology.provision.fog.smithi050:FOG request: PUT /host/155 {"imageID": 6} 2023-06-28T01:42:22.210 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /image {"name": "smithi_ubuntu_22.04"} 2023-06-28T01:42:27.218 INFO:teuthology.provision.fog.smithi050:Scheduling deploy of ubuntu 22.04 2023-06-28T01:42:27.218 DEBUG:teuthology.provision.fog.smithi050:FOG request: GET /task/active None 2023-06-28T01:42:27.219 DEBUG:teuthology.provision.fog.smithi053:Requesting image smithi_ubuntu_22.04 (ID 6) 2023-06-28T01:42:27.220 DEBUG:teuthology.provision.fog.smithi053:FOG request: PUT /host/158 {"imageID": 6} 2023-06-28T01:42:32.293 DEBUG:teuthology.provision.fog.smithi050:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T01:42:32.295 DEBUG:teuthology.provision.fog.smithi083:Requesting image smithi_ubuntu_22.04 (ID 6) 2023-06-28T01:42:32.296 DEBUG:teuthology.provision.fog.smithi083:FOG request: PUT /host/188 {"imageID": 6} 2023-06-28T01:42:32.303 INFO:teuthology.provision.fog.smithi053:Scheduling deploy of ubuntu 22.04 2023-06-28T01:42:32.303 DEBUG:teuthology.provision.fog.smithi053:FOG request: GET /task/active None 2023-06-28T01:42:32.348 DEBUG:teuthology.provision.fog.smithi050:FOG request: POST /host/155/task {"taskTypeID": 1} 2023-06-28T01:42:32.357 INFO:teuthology.provision.fog.smithi083:Scheduling deploy of ubuntu 22.04 2023-06-28T01:42:32.357 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None 2023-06-28T01:42:32.364 DEBUG:teuthology.provision.fog.smithi053:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T01:42:32.398 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T01:42:32.404 DEBUG:teuthology.provision.fog.smithi053:FOG request: POST /host/158/task {"taskTypeID": 1} 2023-06-28T01:42:32.410 DEBUG:teuthology.provision.fog.smithi050:FOG request: GET /task/active None 2023-06-28T01:42:32.442 DEBUG:teuthology.provision.fog.smithi083:FOG request: POST /host/188/task {"taskTypeID": 1} 2023-06-28T01:42:32.456 DEBUG:teuthology.provision.fog.smithi053:FOG request: GET /task/active None 2023-06-28T01:42:37.479 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None 2023-06-28T01:42:47.315 INFO:teuthology.orchestra.console:Power off smithi050 2023-06-28T01:42:47.315 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T01:42:47.329 INFO:teuthology.orchestra.console:Power off smithi053 2023-06-28T01:42:47.330 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T01:42:47.338 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T01:42:47.352 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T01:42:52.400 INFO:teuthology.orchestra.console:Power off smithi083 2023-06-28T01:42:52.401 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T01:42:52.422 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T01:42:52.474 INFO:teuthology.orchestra.console:Power off for smithi050 completed 2023-06-28T01:42:52.475 INFO:teuthology.orchestra.console:Power off for smithi053 completed 2023-06-28T01:42:52.544 INFO:teuthology.orchestra.console:Power off for smithi083 completed 2023-06-28T01:42:52.575 INFO:teuthology.orchestra.console:Power on smithi050 2023-06-28T01:42:52.576 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T01:42:52.585 INFO:teuthology.orchestra.console:Power on smithi053 2023-06-28T01:42:52.585 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T01:42:52.598 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T01:42:52.609 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T01:42:52.645 INFO:teuthology.orchestra.console:Power on smithi083 2023-06-28T01:42:52.646 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T01:42:52.667 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T01:42:56.624 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T01:42:56.633 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T01:42:56.690 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T01:42:56.904 INFO:teuthology.orchestra.console:Power on for smithi083 completed 2023-06-28T01:42:57.005 INFO:teuthology.provision.fog.smithi083:Waiting for deploy to finish 2023-06-28T01:42:57.005 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None 2023-06-28T01:42:57.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T01:43:00.737 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T01:43:00.745 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T01:43:00.948 INFO:teuthology.orchestra.console:Power on for smithi050 completed 2023-06-28T01:43:00.956 INFO:teuthology.orchestra.console:Power on for smithi053 completed 2023-06-28T01:43:01.050 INFO:teuthology.provision.fog.smithi050:Waiting for deploy to finish 2023-06-28T01:43:01.050 DEBUG:teuthology.provision.fog.smithi050:FOG request: GET /task/active None 2023-06-28T01:43:06.056 INFO:teuthology.provision.fog.smithi053:Waiting for deploy to finish 2023-06-28T01:43:06.057 DEBUG:teuthology.provision.fog.smithi053:FOG request: GET /task/active None 2023-06-28T01:43:07.736 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T01:43:15.568 WARNING:teuthology.provision.fog.smithi083:[Errno None] Unable to connect to port 22 on 172.21.15.83 2023-06-28T01:43:21.576 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T01:43:22.750 DEBUG:teuthology.provision.fog.smithi050:FOG request: GET /task/active None 2023-06-28T01:43:27.757 WARNING:teuthology.provision.fog.smithi053:[Errno None] Unable to connect to port 22 on 172.21.15.53 2023-06-28T01:43:27.757 WARNING:teuthology.provision.fog.smithi083:[Errno None] Unable to connect to port 22 on 172.21.15.83 2023-06-28T01:43:33.764 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T01:43:38.770 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T01:43:43.777 DEBUG:teuthology.provision.fog.smithi050:FOG request: GET /task/active None 2023-06-28T01:43:43.778 WARNING:teuthology.provision.fog.smithi053:[Errno None] Unable to connect to port 22 on 172.21.15.53 2023-06-28T01:43:48.782 WARNING:teuthology.provision.fog.smithi083:[Errno None] Unable to connect to port 22 on 172.21.15.83 2023-06-28T01:43:49.781 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T01:43:52.848 WARNING:teuthology.provision.fog.smithi053:[Errno None] Unable to connect to port 22 on 172.21.15.53 2023-06-28T01:43:54.785 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T01:43:57.840 WARNING:teuthology.provision.fog.smithi083:[Errno None] Unable to connect to port 22 on 172.21.15.83 2023-06-28T01:43:58.850 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T01:44:03.844 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T01:44:03.861 DEBUG:teuthology.provision.fog.smithi050:FOG request: GET /task/active None 2023-06-28T01:44:08.865 WARNING:teuthology.provision.fog.smithi083:[Errno None] Unable to connect to port 22 on 172.21.15.83 2023-06-28T01:44:08.866 WARNING:teuthology.provision.fog.smithi053:[Errno None] Unable to connect to port 22 on 172.21.15.53 2023-06-28T01:44:14.872 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T01:44:14.875 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T01:44:17.936 WARNING:teuthology.provision.fog.smithi053:[Errno None] Unable to connect to port 22 on 172.21.15.53 2023-06-28T01:44:20.980 WARNING:teuthology.provision.fog.smithi083:[Errno None] Unable to connect to port 22 on 172.21.15.83 2023-06-28T01:44:23.936 DEBUG:teuthology.provision.fog.smithi050:FOG request: GET /task/active None 2023-06-28T01:44:28.941 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T01:44:33.947 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T01:44:39.022 ERROR:teuthology.orchestra.connection:Error authenticating with smithi053.front.sepia.ceph.com: Authentication failed. 2023-06-28T01:44:39.024 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-06-28T01:44:39.025 DEBUG:teuthology.orchestra.run.smithi053:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-06-28T01:44:39.025 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:39.070 ERROR:teuthology.orchestra.connection:Error authenticating with smithi083.front.sepia.ceph.com: Authentication failed. 2023-06-28T01:44:39.070 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-06-28T01:44:39.071 DEBUG:teuthology.orchestra.run.smithi083:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-06-28T01:44:39.071 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:40.032 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:40.077 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:41.039 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:41.084 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:42.047 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:42.092 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:43.054 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:43.099 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:44.060 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:44.106 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:45.067 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:45.112 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:46.075 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:46.120 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:47.083 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:47.128 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:48.090 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:48.135 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:49.098 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:49.143 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:50.107 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:50.152 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:51.115 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:51.160 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:52.123 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:52.168 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:53.130 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:53.175 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:54.042 DEBUG:teuthology.provision.fog.smithi050:FOG request: GET /task/active None 2023-06-28T01:44:59.049 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:44:59.049 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:00.055 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:00.055 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:01.060 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:01.061 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:02.066 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:02.067 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:03.073 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:03.073 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:04.078 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:04.079 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:05.084 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:05.084 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:06.090 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:06.090 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:07.095 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:07.096 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:08.101 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:08.101 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:09.106 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:09.107 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:10.111 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:10.112 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:11.117 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:11.118 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:12.123 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:12.124 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:13.129 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:13.130 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:14.135 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:14.135 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:14.142 DEBUG:teuthology.provision.fog.smithi050:FOG request: GET /task/active None 2023-06-28T01:45:15.140 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:15.140 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:16.152 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:16.152 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:17.157 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:17.158 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:18.168 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:18.169 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:19.174 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:19.174 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:20.185 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:20.186 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:21.191 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:21.191 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:22.201 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:22.202 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:23.207 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:23.208 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:24.214 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:24.215 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:25.219 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:25.219 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:26.223 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:26.223 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:27.228 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:27.258 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:28.263 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:28.264 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:29.235 DEBUG:teuthology.provision.fog.smithi050:FOG request: GET /task/active None 2023-06-28T01:45:34.242 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:34.243 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:35.246 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:35.247 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:36.252 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:36.253 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:37.257 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:37.258 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:38.262 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:38.263 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:39.267 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:39.267 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:40.272 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:40.272 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:41.276 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:41.277 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:42.281 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:42.282 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:43.286 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:43.287 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:44.291 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:44.291 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:45.296 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:45.296 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:46.302 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:46.302 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:47.308 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:47.308 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:48.314 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:48.314 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:49.319 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:49.320 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:49.337 DEBUG:teuthology.provision.fog.smithi050:FOG request: GET /task/active None 2023-06-28T01:45:50.325 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:50.326 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:51.332 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:51.333 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:52.339 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:53.345 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:54.351 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:55.358 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:56.364 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:57.371 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:58.377 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:45:59.384 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:46:00.391 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:46:01.398 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:46:02.405 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:46:03.412 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:46:04.419 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T01:46:04.431 DEBUG:teuthology.provision.fog.smithi050:FOG request: GET /task/active None 2023-06-28T01:46:09.504 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T01:46:27.828 WARNING:teuthology.provision.fog.smithi050:[Errno None] Unable to connect to port 22 on 172.21.15.50 2023-06-28T01:46:33.830 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T01:46:36.880 WARNING:teuthology.provision.fog.smithi050:[Errno None] Unable to connect to port 22 on 172.21.15.50 2023-06-28T01:46:42.882 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T01:47:39.504 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2023-06-28T01:47:39.507 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/teuthology/provision/fog.py", line 295, in _wait_for_ready self.remote.run(args=cmd, timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/teuthology/orchestra/run.py", line 97, in execute self.client.exec_command(self.command, timeout=self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 510, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 919, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1050, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2109, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) ConnectionResetError: [Errno 104] Connection reset by peer 2023-06-28T01:47:39.508 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-28T01:47:39.508 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-28T01:47:44.590 INFO:teuthology.nuke:Checking targets against current locks 2023-06-28T01:47:47.050 DEBUG:teuthology.nuke:shortname: smithi050 2023-06-28T01:47:47.051 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T01:47:47.072 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-27_20:22:12-rbd-wip-yuri11-testing-2023-06-27-0812-reef-distro-default-smithi/7318585', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 01:41:37.851075', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0='} 2023-06-28T01:47:47.074 DEBUG:teuthology.nuke:shortname: smithi053 2023-06-28T01:47:47.075 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T01:47:47.096 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-27_20:22:12-rbd-wip-yuri11-testing-2023-06-27-0812-reef-distro-default-smithi/7318585', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 01:41:37.849833', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-06-28T01:47:47.097 DEBUG:teuthology.nuke:shortname: smithi083 2023-06-28T01:47:47.097 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T01:47:47.117 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-27_20:22:12-rbd-wip-yuri11-testing-2023-06-27-0812-reef-distro-default-smithi/7318585', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 01:41:37.847781', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-06-28T01:47:47.125 INFO:teuthology.orchestra.console:Power off smithi050 2023-06-28T01:47:47.126 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T01:47:47.148 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T01:47:47.164 INFO:teuthology.orchestra.console:Power off smithi053 2023-06-28T01:47:47.165 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T01:47:47.187 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T01:47:47.203 INFO:teuthology.orchestra.console:Power off smithi083 2023-06-28T01:47:47.204 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T01:47:47.224 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T01:47:47.886 WARNING:teuthology.provision.fog.smithi050:timed out 2023-06-28T01:47:51.179 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T01:47:51.214 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T01:47:51.250 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T01:47:53.889 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T01:47:58.896 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T01:47:58.905 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T01:47:58.914 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T01:48:04.009 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T01:48:04.018 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T01:48:04.028 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T01:48:04.220 INFO:teuthology.orchestra.console:Power off for smithi050 completed 2023-06-28T01:48:04.239 INFO:teuthology.orchestra.console:Power off for smithi083 completed 2023-06-28T01:48:06.309 INFO:teuthology.lock.ops:unlocked: smithi050.front.sepia.ceph.com 2023-06-28T01:48:06.318 INFO:teuthology.lock.ops:unlocked: smithi083.front.sepia.ceph.com 2023-06-28T01:48:10.249 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T01:48:15.361 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T01:48:17.392 WARNING:teuthology.provision.fog.smithi050:[Errno None] Unable to connect to port 22 on 172.21.15.50 2023-06-28T01:48:20.471 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T01:48:20.683 INFO:teuthology.orchestra.console:Power off for smithi053 completed 2023-06-28T01:48:25.789 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T01:48:35.825 WARNING:teuthology.provision.fog.smithi050:[Errno None] Unable to connect to port 22 on 172.21.15.50 2023-06-28T01:48:35.856 INFO:teuthology.lock.ops:unlocked: smithi053.front.sepia.ceph.com 2023-06-28T01:48:45.994 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T01:48:56.129 WARNING:teuthology.provision.fog.smithi050:[Errno None] Unable to connect to port 22 on 172.21.15.50 2023-06-28T01:48:56.255 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/teuthology/dispatcher/supervisor.py", line 57, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/teuthology/provision/fog.py", line 295, in _wait_for_ready self.remote.run(args=cmd, timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/teuthology/orchestra/run.py", line 97, in execute self.client.exec_command(self.command, timeout=self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 510, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 919, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1050, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2109, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_3fdf1f2d90b4263fafe8c0792229f37d37a2ab95/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) ConnectionResetError: [Errno 104] Connection reset by peer