2023-06-28T14:50:47.635 INFO:root:teuthology version: 0.0.1.dev132+gaa9345a 2023-06-28T14:50:47.636 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-28T14:50:47.637 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-28T14:50:52.087 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f216ba41430>, signals=[15]) 2023-06-28T14:50:52.088 INFO:teuthology.lock.ops:Start node 'smithi088.front.sepia.ceph.com' reimaging 2023-06-28T14:50:52.089 INFO:teuthology.lock.ops:Updating [smithi088.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T14:50:52.089 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server 2023-06-28T14:50:52.112 INFO:teuthology.lock.ops:Start node 'smithi132.front.sepia.ceph.com' reimaging 2023-06-28T14:50:52.112 INFO:teuthology.lock.ops:Updating [smithi132.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T14:50:52.113 INFO:teuthology.lock.ops:Updating smithi132.front.sepia.ceph.com on lock server 2023-06-28T14:50:52.114 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /host {"name": "smithi088"} 2023-06-28T14:51:07.136 INFO:teuthology.lock.ops:Start node 'smithi154.front.sepia.ceph.com' reimaging 2023-06-28T14:51:07.137 INFO:teuthology.lock.ops:Updating [smithi154.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T14:51:07.137 INFO:teuthology.lock.ops:Updating smithi154.front.sepia.ceph.com on lock server 2023-06-28T14:51:07.138 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /host {"name": "smithi132"} 2023-06-28T14:51:12.027 DEBUG:teuthology.provision.fog.smithi154:FOG request: GET /host {"name": "smithi154"} 2023-06-28T14:51:12.045 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /image {"name": "smithi_centos_8.stream"} 2023-06-28T14:51:12.069 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /image {"name": "smithi_centos_8.stream"} 2023-06-28T14:51:12.096 DEBUG:teuthology.provision.fog.smithi154:FOG request: GET /image {"name": "smithi_centos_8.stream"} 2023-06-28T14:51:12.098 DEBUG:teuthology.provision.fog.smithi088:Requesting image smithi_centos_8.stream (ID 3) 2023-06-28T14:51:12.099 DEBUG:teuthology.provision.fog.smithi088:FOG request: PUT /host/193 {"imageID": 3} 2023-06-28T14:51:12.112 DEBUG:teuthology.provision.fog.smithi132:Requesting image smithi_centos_8.stream (ID 3) 2023-06-28T14:51:12.113 DEBUG:teuthology.provision.fog.smithi132:FOG request: PUT /host/237 {"imageID": 3} 2023-06-28T14:51:12.147 DEBUG:teuthology.provision.fog.smithi154:Requesting image smithi_centos_8.stream (ID 3) 2023-06-28T14:51:12.147 DEBUG:teuthology.provision.fog.smithi154:FOG request: PUT /host/259 {"imageID": 3} 2023-06-28T14:51:12.153 INFO:teuthology.provision.fog.smithi088:Scheduling deploy of centos 8.stream 2023-06-28T14:51:12.153 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None 2023-06-28T14:51:12.169 INFO:teuthology.provision.fog.smithi132:Scheduling deploy of centos 8.stream 2023-06-28T14:51:12.169 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None 2023-06-28T14:51:12.197 INFO:teuthology.provision.fog.smithi154:Scheduling deploy of centos 8.stream 2023-06-28T14:51:12.197 DEBUG:teuthology.provision.fog.smithi154:FOG request: GET /task/active None 2023-06-28T14:51:12.231 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T14:51:12.249 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T14:51:12.273 DEBUG:teuthology.provision.fog.smithi154:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T14:51:12.275 DEBUG:teuthology.provision.fog.smithi088:FOG request: POST /host/193/task {"taskTypeID": 1} 2023-06-28T14:51:12.291 DEBUG:teuthology.provision.fog.smithi132:FOG request: POST /host/237/task {"taskTypeID": 1} 2023-06-28T14:51:12.320 DEBUG:teuthology.provision.fog.smithi154:FOG request: POST /host/259/task {"taskTypeID": 1} 2023-06-28T14:51:12.331 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None 2023-06-28T14:51:12.385 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None 2023-06-28T14:51:12.400 DEBUG:teuthology.provision.fog.smithi154:FOG request: GET /task/active None 2023-06-28T14:51:12.470 INFO:teuthology.orchestra.console:Power off smithi088 2023-06-28T14:51:12.471 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T14:51:12.494 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:51:12.523 INFO:teuthology.orchestra.console:Power off smithi132 2023-06-28T14:51:12.524 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T14:51:12.538 INFO:teuthology.orchestra.console:Power off smithi154 2023-06-28T14:51:12.539 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T14:51:12.546 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:51:12.559 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:51:12.682 INFO:teuthology.orchestra.console:Power off for smithi154 completed 2023-06-28T14:51:12.783 INFO:teuthology.orchestra.console:Power on smithi154 2023-06-28T14:51:12.784 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T14:51:12.806 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:51:16.548 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:51:16.571 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:51:16.828 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:51:21.662 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:51:21.682 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:51:21.893 INFO:teuthology.orchestra.console:Power off for smithi132 completed 2023-06-28T14:51:21.941 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:51:21.994 INFO:teuthology.orchestra.console:Power on smithi132 2023-06-28T14:51:21.995 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T14:51:22.021 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:51:26.053 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:51:26.312 INFO:teuthology.orchestra.console:Power on for smithi132 completed 2023-06-28T14:51:26.413 INFO:teuthology.provision.fog.smithi132:Waiting for deploy to finish 2023-06-28T14:51:26.414 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None 2023-06-28T14:51:31.421 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:51:31.430 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:51:36.533 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:51:36.540 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:51:41.643 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:51:41.651 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:51:46.516 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None 2023-06-28T14:51:51.525 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:51:51.533 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:51:56.636 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:51:56.647 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:52:01.751 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:52:01.760 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:52:01.973 INFO:teuthology.orchestra.console:Power on for smithi154 completed 2023-06-28T14:52:02.074 INFO:teuthology.provision.fog.smithi154:Waiting for deploy to finish 2023-06-28T14:52:02.075 DEBUG:teuthology.provision.fog.smithi154:FOG request: GET /task/active None 2023-06-28T14:52:07.082 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None 2023-06-28T14:52:11.088 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:52:16.203 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:52:21.316 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:52:22.699 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None 2023-06-28T14:52:27.319 DEBUG:teuthology.provision.fog.smithi154:FOG request: GET /task/active None 2023-06-28T14:52:27.321 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:52:27.536 INFO:teuthology.orchestra.console:Power off for smithi088 completed 2023-06-28T14:52:27.637 INFO:teuthology.orchestra.console:Power on smithi088 2023-06-28T14:52:27.637 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T14:52:27.663 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:52:31.692 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:52:36.805 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T14:52:37.015 INFO:teuthology.orchestra.console:Power on for smithi088 completed 2023-06-28T14:52:37.116 INFO:teuthology.provision.fog.smithi088:Waiting for deploy to finish 2023-06-28T14:52:37.117 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None 2023-06-28T14:52:42.450 DEBUG:teuthology.provision.fog.smithi154:FOG request: GET /task/active None 2023-06-28T14:52:47.457 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None 2023-06-28T14:52:47.572 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T14:52:57.216 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None 2023-06-28T14:53:02.576 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None 2023-06-28T14:53:07.581 WARNING:teuthology.provision.fog.smithi154:[Errno None] Unable to connect to port 22 on 172.21.15.154 2023-06-28T14:53:13.588 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T14:53:16.660 WARNING:teuthology.provision.fog.smithi154:[Errno None] Unable to connect to port 22 on 172.21.15.154 2023-06-28T14:53:17.298 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None 2023-06-28T14:53:22.648 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None 2023-06-28T14:53:27.654 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T14:53:30.704 WARNING:teuthology.provision.fog.smithi154:[Errno None] Unable to connect to port 22 on 172.21.15.154 2023-06-28T14:53:32.379 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None 2023-06-28T14:53:37.385 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T14:53:42.737 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None 2023-06-28T14:53:47.745 WARNING:teuthology.provision.fog.smithi154:[Errno None] Unable to connect to port 22 on 172.21.15.154 2023-06-28T14:53:53.748 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T14:53:57.459 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None 2023-06-28T14:54:00.240 WARNING:teuthology.provision.fog.smithi154:[Errno None] Unable to connect to port 22 on 172.21.15.154 2023-06-28T14:54:02.827 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None 2023-06-28T14:54:06.244 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T14:54:14.192 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None 2023-06-28T14:54:19.197 WARNING:teuthology.provision.fog.smithi154:[Errno None] Unable to connect to port 22 on 172.21.15.154 2023-06-28T14:54:19.403 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None 2023-06-28T14:54:24.473 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T14:54:29.478 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T14:54:34.484 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None 2023-06-28T14:54:47.920 WARNING:teuthology.provision.fog.smithi132:[Errno None] Unable to connect to port 22 on 172.21.15.132 2023-06-28T14:54:53.040 WARNING:teuthology.provision.fog.smithi154:[Errno None] Unable to connect to port 22 on 172.21.15.154 2023-06-28T14:54:53.923 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T14:54:58.928 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None 2023-06-28T14:55:03.934 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T14:55:08.937 WARNING:teuthology.provision.fog.smithi132:[Errno None] Unable to connect to port 22 on 172.21.15.132 2023-06-28T14:55:09.047 ERROR:teuthology.orchestra.connection:Error authenticating with smithi154.front.sepia.ceph.com: Authentication failed. 2023-06-28T14:55:09.048 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-06-28T14:55:09.049 DEBUG:teuthology.orchestra.run.smithi154:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-06-28T14:55:09.050 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:10.055 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:11.059 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:12.066 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:13.069 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:14.073 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:14.939 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T14:55:19.946 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:20.950 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:21.954 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:22.960 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:23.024 WARNING:teuthology.provision.fog.smithi132:[Errno None] Unable to connect to port 22 on 172.21.15.132 2023-06-28T14:55:23.965 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:24.035 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None 2023-06-28T14:55:29.041 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T14:55:29.044 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:29.150 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T14:55:30.050 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:31.054 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:32.060 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:33.065 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:34.070 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:35.076 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:36.081 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:37.087 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:38.092 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:39.098 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:40.104 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:41.109 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:42.115 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:43.121 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:44.127 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:45.133 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:46.138 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:47.143 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:47.572 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88 2023-06-28T14:55:48.149 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:49.154 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:50.160 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:51.166 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:52.172 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:53.178 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:53.574 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T14:55:58.578 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:55:59.582 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:00.587 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:00.852 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-06-28T14:56:00.853 DEBUG:teuthology.orchestra.run.smithi132:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-06-28T14:56:01.600 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:01.648 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88 2023-06-28T14:56:02.607 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:03.615 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:04.623 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:05.632 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:06.641 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:07.649 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T14:56:08.998 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:09.099 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:10.143 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:11.149 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:12.048 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88 2023-06-28T14:56:12.155 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:13.161 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:14.170 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:15.177 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:16.183 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:17.189 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:18.049 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T14:56:23.053 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:24.058 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:25.062 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:25.876 INFO:teuthology.provision.fog.smithi132:Node is ready 2023-06-28T14:56:25.876 DEBUG:teuthology.orchestra.run.smithi132:> hostname 2023-06-28T14:56:25.901 INFO:teuthology.orchestra.run.smithi132.stdout:smithi132.front.sepia.ceph.com 2023-06-28T14:56:25.901 DEBUG:teuthology.orchestra.run.smithi132:> grep smithi132.front.sepia.ceph.com /etc/hosts 2023-06-28T14:56:25.962 INFO:teuthology.orchestra.run.smithi132.stdout:172.21.15.132 smithi132.front.sepia.ceph.com smithi132 2023-06-28T14:56:25.962 DEBUG:teuthology.orchestra.run.smithi132:> sudo hostname smithi132 2023-06-28T14:56:26.069 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:26.078 DEBUG:teuthology.orchestra.run.smithi132:> sudo sed -i -e 's/smithi132.front.sepia.ceph.com/smithi132/g' /etc/hosts 2023-06-28T14:56:26.111 DEBUG:teuthology.orchestra.run.smithi132:> sudo sed -i -e 's/172.21.15.132/172.21.15.132/g' /etc/hosts 2023-06-28T14:56:26.128 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88 2023-06-28T14:56:26.180 DEBUG:teuthology.orchestra.run.smithi132:> sudo sed -i -e 's/smithi132.front.sepia.ceph.com/smithi132/g' /etc/hostname 2023-06-28T14:56:26.248 DEBUG:teuthology.orchestra.run.smithi132:> sudo hostname smithi132 2023-06-28T14:56:26.316 DEBUG:teuthology.orchestra.run.smithi132:> cat /etc/os-release 2023-06-28T14:56:26.380 INFO:teuthology.orchestra.run.smithi132.stdout:NAME="CentOS Stream" 2023-06-28T14:56:26.380 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION="8" 2023-06-28T14:56:26.381 INFO:teuthology.orchestra.run.smithi132.stdout:ID="centos" 2023-06-28T14:56:26.381 INFO:teuthology.orchestra.run.smithi132.stdout:ID_LIKE="rhel fedora" 2023-06-28T14:56:26.381 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION_ID="8" 2023-06-28T14:56:26.381 INFO:teuthology.orchestra.run.smithi132.stdout:PLATFORM_ID="platform:el8" 2023-06-28T14:56:26.381 INFO:teuthology.orchestra.run.smithi132.stdout:PRETTY_NAME="CentOS Stream 8" 2023-06-28T14:56:26.381 INFO:teuthology.orchestra.run.smithi132.stdout:ANSI_COLOR="0;31" 2023-06-28T14:56:26.381 INFO:teuthology.orchestra.run.smithi132.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-06-28T14:56:26.382 INFO:teuthology.orchestra.run.smithi132.stdout:HOME_URL="https://centos.org/" 2023-06-28T14:56:26.382 INFO:teuthology.orchestra.run.smithi132.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-28T14:56:26.382 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-28T14:56:26.382 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-06-28T14:56:26.383 INFO:teuthology.provision.fog.smithi132:Deploy complete! 2023-06-28T14:56:26.384 DEBUG:teuthology.parallel:result is None 2023-06-28T14:56:27.075 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:28.081 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:29.086 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:30.092 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:31.098 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:32.103 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:32.129 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T14:56:37.134 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:38.138 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:39.142 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:40.147 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:41.153 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:42.158 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:43.164 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:44.169 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:45.175 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:46.180 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:47.187 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:48.193 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:49.199 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:50.204 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:51.210 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:52.215 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:53.221 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:54.227 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:55.233 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:56.240 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:56:57.245 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T14:57:37.133 WARNING:teuthology.provision.fog.smithi088:timed out 2023-06-28T14:57:43.136 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T14:57:43.392 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-06-28T14:57:43.393 DEBUG:teuthology.orchestra.run.smithi088:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-06-28T14:57:43.677 INFO:teuthology.provision.fog.smithi088:Node is ready 2023-06-28T14:57:43.677 DEBUG:teuthology.orchestra.run.smithi088:> hostname 2023-06-28T14:57:43.736 INFO:teuthology.orchestra.run.smithi088.stdout:smithi088.front.sepia.ceph.com 2023-06-28T14:57:43.736 DEBUG:teuthology.orchestra.run.smithi088:> grep smithi088.front.sepia.ceph.com /etc/hosts 2023-06-28T14:57:43.795 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88 smithi088.front.sepia.ceph.com smithi088 2023-06-28T14:57:43.795 DEBUG:teuthology.orchestra.run.smithi088:> sudo hostname smithi088 2023-06-28T14:57:43.897 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i -e 's/smithi088.front.sepia.ceph.com/smithi088/g' /etc/hosts 2023-06-28T14:57:43.926 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i -e 's/172.21.15.88/172.21.15.88/g' /etc/hosts 2023-06-28T14:57:43.993 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i -e 's/smithi088.front.sepia.ceph.com/smithi088/g' /etc/hostname 2023-06-28T14:57:44.061 DEBUG:teuthology.orchestra.run.smithi088:> sudo hostname smithi088 2023-06-28T14:57:44.127 DEBUG:teuthology.orchestra.run.smithi088:> cat /etc/os-release 2023-06-28T14:57:44.184 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="CentOS Stream" 2023-06-28T14:57:44.185 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="8" 2023-06-28T14:57:44.185 INFO:teuthology.orchestra.run.smithi088.stdout:ID="centos" 2023-06-28T14:57:44.185 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE="rhel fedora" 2023-06-28T14:57:44.185 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="8" 2023-06-28T14:57:44.185 INFO:teuthology.orchestra.run.smithi088.stdout:PLATFORM_ID="platform:el8" 2023-06-28T14:57:44.185 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="CentOS Stream 8" 2023-06-28T14:57:44.186 INFO:teuthology.orchestra.run.smithi088.stdout:ANSI_COLOR="0;31" 2023-06-28T14:57:44.186 INFO:teuthology.orchestra.run.smithi088.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-06-28T14:57:44.186 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://centos.org/" 2023-06-28T14:57:44.186 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-28T14:57:44.186 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-28T14:57:44.186 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-06-28T14:57:44.187 INFO:teuthology.provision.fog.smithi088:Deploy complete! 2023-06-28T14:57:44.187 DEBUG:teuthology.parallel:result is None 2023-06-28T15:02:45.617 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2023-06-28T15:02:45.621 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 295, in _wait_for_ready self.remote.run(args=cmd, timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 97, in execute self.client.exec_command(self.command, timeout=self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 510, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 919, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1050, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2109, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) ConnectionResetError: [Errno 104] Connection reset by peer 2023-06-28T15:02:45.623 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-28T15:02:45.624 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-28T15:02:50.723 INFO:teuthology.nuke:Checking targets against current locks 2023-06-28T15:03:05.794 DEBUG:teuthology.nuke:shortname: smithi088 2023-06-28T15:03:05.794 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T15:03:09.532 DEBUG:teuthology.nuke:shortname: smithi132 2023-06-28T15:03:09.532 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T15:03:09.536 INFO:teuthology.nuke:Not nuking smithi154.front.sepia.ceph.com because description doesn't match: yuriw-2023-06-28_14:13:04-rados-quincy_release-distro-default-smithi != /home/teuthworker/archive/matan-2023-06-28_14:58:01-crimson-rados-wip-matanb-crimson-bug-61651-logs-distro-crimson-smithi/7320503 2023-06-28T15:03:09.538 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-28_14:13:04-rados-quincy_release-distro-default-smithi/7320230', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 14:50:26.754396', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9tDeP7UnqUkCXqkuJJiSvm+8g13nObL1LQNtUYR9s8uvRQAh4NrzD+3RCB1xjM7SPSCSpPiQdjuZEAbxKDTDz1hgWRKHiBIDIpR6X+l1z3RArc3BtQ7xxtyz7oTOGdW6pzzOES3oVy+5McqV4UCIrYeVVrXFD6FRHDjy1KVoQnKSe5B/PQ9rYH3am5YVn61VGg4gRr2Q8Kw0JD1vG2JZbA0qfbVb0lESNe2Y5rVq6Do16oT6M6dKc2RgG4oGB48C+kds91N3yrbcC/ZBF/v9ZPZNzxX74fRfYsgRPS06igpMFPiOD75zrp/FeLzvePn6gspwU0c3IxKgJPCOe6ANNWtlrAEUrMJU/uefux0TlCBQx1HODR4iVEgZjT7bGB0EttLQHRYb7B1K1KVMXo42JY7OoXKXUT8s9UBMWeZ8T9zNKxwBqnVUpwVI06o+vg8/AcL/YRA4/oRIyBGAIsFBPT10qBdReXQNyZCdoXPmlnwDeO/yEPgL5u8klg9RpXak='} 2023-06-28T15:03:09.562 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-28_14:13:04-rados-quincy_release-distro-default-smithi/7320230', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 14:50:26.757198', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLrBspCz8+H6fuPQTDXq6l8I4QBR2cMUpxF8lP95+nf5QNz3G8LPA18DzDH1HiaeWR2atg7gjdONiye1S8N2FWQ='} 2023-06-28T15:03:09.597 INFO:teuthology.orchestra.console:Power off smithi088 2023-06-28T15:03:09.598 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T15:03:09.637 INFO:teuthology.orchestra.console:Power off smithi132 2023-06-28T15:03:09.638 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T15:03:09.650 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:03:09.667 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:03:13.685 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:03:13.698 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:03:18.797 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:03:18.810 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:03:23.909 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:03:23.923 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:03:29.024 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:03:29.033 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:03:34.137 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:03:34.144 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:03:39.249 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:03:39.258 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:03:39.461 INFO:teuthology.orchestra.console:Power off for smithi088 completed 2023-06-28T15:03:41.648 INFO:teuthology.lock.ops:unlocked: smithi088.front.sepia.ceph.com 2023-06-28T15:03:45.600 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:03:50.712 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:03:55.828 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:04:00.938 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:04:01.150 INFO:teuthology.orchestra.console:Power off for smithi132 completed 2023-06-28T15:04:11.011 INFO:teuthology.lock.ops:unlocked: smithi132.front.sepia.ceph.com 2023-06-28T15:04:11.013 ERROR:teuthology.nuke:Could not nuke the following targets: targets: smithi154.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= 2023-06-28T15:04:11.394 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 57, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 295, in _wait_for_ready self.remote.run(args=cmd, timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 97, in execute self.client.exec_command(self.command, timeout=self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 510, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 919, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1050, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2109, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) ConnectionResetError: [Errno 104] Connection reset by peer