2023-06-28T07:14:46.464 INFO:root:teuthology version: 0.0.1.dev132+gaa9345a 2023-06-28T07:14:46.466 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-28T07:14:46.466 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-28T07:15:16.671 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd6ff7583a0>, signals=[15]) 2023-06-28T07:15:16.673 INFO:teuthology.lock.ops:Start node 'smithi150.front.sepia.ceph.com' reimaging 2023-06-28T07:15:16.674 INFO:teuthology.lock.ops:Updating [smithi150.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T07:15:16.674 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2023-06-28T07:15:21.709 INFO:teuthology.lock.ops:Start node 'smithi156.front.sepia.ceph.com' reimaging 2023-06-28T07:15:21.710 INFO:teuthology.lock.ops:Updating [smithi156.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T07:15:21.710 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server 2023-06-28T07:15:21.711 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /host {"name": "smithi150"} 2023-06-28T07:15:31.749 INFO:teuthology.lock.ops:Start node 'smithi195.front.sepia.ceph.com' reimaging 2023-06-28T07:15:31.749 INFO:teuthology.lock.ops:Updating [smithi195.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T07:15:31.749 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2023-06-28T07:15:31.750 DEBUG:teuthology.provision.fog.smithi156:FOG request: GET /host {"name": "smithi156"} 2023-06-28T07:15:41.060 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /host {"name": "smithi195"} 2023-06-28T07:15:51.070 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /image {"name": "smithi_ubuntu_20.04"} 2023-06-28T07:16:01.082 DEBUG:teuthology.provision.fog.smithi156:FOG request: GET /image {"name": "smithi_ubuntu_20.04"} 2023-06-28T07:16:01.111 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /image {"name": "smithi_ubuntu_20.04"} 2023-06-28T07:16:01.126 DEBUG:teuthology.provision.fog.smithi156:Requesting image smithi_ubuntu_20.04 (ID 2) 2023-06-28T07:16:01.126 DEBUG:teuthology.provision.fog.smithi156:FOG request: PUT /host/261 {"imageID": 2} 2023-06-28T07:16:01.129 DEBUG:teuthology.provision.fog.smithi150:Requesting image smithi_ubuntu_20.04 (ID 2) 2023-06-28T07:16:01.129 DEBUG:teuthology.provision.fog.smithi150:FOG request: PUT /host/255 {"imageID": 2} 2023-06-28T07:16:01.160 DEBUG:teuthology.provision.fog.smithi195:Requesting image smithi_ubuntu_20.04 (ID 2) 2023-06-28T07:16:01.161 DEBUG:teuthology.provision.fog.smithi195:FOG request: PUT /host/300 {"imageID": 2} 2023-06-28T07:16:01.174 INFO:teuthology.provision.fog.smithi150:Scheduling deploy of ubuntu 20.04 2023-06-28T07:16:01.174 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None 2023-06-28T07:16:01.177 INFO:teuthology.provision.fog.smithi156:Scheduling deploy of ubuntu 20.04 2023-06-28T07:16:01.177 DEBUG:teuthology.provision.fog.smithi156:FOG request: GET /task/active None 2023-06-28T07:16:01.206 INFO:teuthology.provision.fog.smithi195:Scheduling deploy of ubuntu 20.04 2023-06-28T07:16:01.207 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /task/active None 2023-06-28T07:16:01.228 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T07:16:01.234 DEBUG:teuthology.provision.fog.smithi156:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T07:16:01.248 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T07:16:01.266 DEBUG:teuthology.provision.fog.smithi150:FOG request: POST /host/255/task {"taskTypeID": 1} 2023-06-28T07:16:01.270 DEBUG:teuthology.provision.fog.smithi156:FOG request: POST /host/261/task {"taskTypeID": 1} 2023-06-28T07:16:01.287 DEBUG:teuthology.provision.fog.smithi195:FOG request: POST /host/300/task {"taskTypeID": 1} 2023-06-28T07:16:01.324 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None 2023-06-28T07:16:01.335 DEBUG:teuthology.provision.fog.smithi156:FOG request: GET /task/active None 2023-06-28T07:16:01.357 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /task/active None 2023-06-28T07:16:11.785 INFO:teuthology.orchestra.console:Power off smithi195 2023-06-28T07:16:11.786 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T07:16:11.802 INFO:teuthology.orchestra.console:Power off smithi156 2023-06-28T07:16:11.804 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T07:16:11.820 INFO:teuthology.orchestra.console:Power off smithi150 2023-06-28T07:16:11.821 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T07:16:11.832 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:11.840 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:11.850 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:11.962 INFO:teuthology.orchestra.console:Power off for smithi156 completed 2023-06-28T07:16:11.973 INFO:teuthology.orchestra.console:Power off for smithi150 completed 2023-06-28T07:16:12.064 INFO:teuthology.orchestra.console:Power on smithi156 2023-06-28T07:16:12.065 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T07:16:12.075 INFO:teuthology.orchestra.console:Power on smithi150 2023-06-28T07:16:12.075 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T07:16:12.086 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:12.096 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:15.871 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:16.108 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:16.119 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:19.986 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:20.222 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:20.231 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:25.099 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:25.334 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:25.343 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:30.210 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:30.422 INFO:teuthology.orchestra.console:Power off for smithi195 completed 2023-06-28T07:16:30.446 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:30.454 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:30.523 INFO:teuthology.orchestra.console:Power on smithi195 2023-06-28T07:16:30.523 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T07:16:30.543 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:30.663 INFO:teuthology.orchestra.console:Power on for smithi150 completed 2023-06-28T07:16:30.764 INFO:teuthology.provision.fog.smithi150:Waiting for deploy to finish 2023-06-28T07:16:30.765 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None 2023-06-28T07:16:34.568 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:35.557 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:38.680 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:40.669 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:43.791 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:45.779 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:45.822 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None 2023-06-28T07:16:48.901 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:53.396 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:54.011 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:58.511 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:16:58.722 INFO:teuthology.orchestra.console:Power on for smithi156 completed 2023-06-28T07:16:58.824 INFO:teuthology.provision.fog.smithi156:Waiting for deploy to finish 2023-06-28T07:16:58.824 DEBUG:teuthology.provision.fog.smithi156:FOG request: GET /task/active None 2023-06-28T07:16:58.903 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:16:59.122 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:17:03.464 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None 2023-06-28T07:17:08.473 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:17:13.587 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:17:17.296 WARNING:teuthology.provision.fog.smithi156:[Errno None] Unable to connect to port 22 on 172.21.15.156 2023-06-28T07:17:18.698 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:17:23.301 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:17:28.307 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None 2023-06-28T07:17:28.309 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:17:33.621 INFO:teuthology.orchestra.console:Power on for smithi195 completed 2023-06-28T07:17:33.722 INFO:teuthology.provision.fog.smithi195:Waiting for deploy to finish 2023-06-28T07:17:33.722 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /task/active None 2023-06-28T07:17:33.792 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:17:46.736 WARNING:teuthology.provision.fog.smithi156:[Errno None] Unable to connect to port 22 on 172.21.15.156 2023-06-28T07:17:48.402 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None 2023-06-28T07:17:53.407 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:17:58.527 ERROR:teuthology.orchestra.connection:Error authenticating with smithi156.front.sepia.ceph.com: Authentication failed. 2023-06-28T07:17:58.529 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-06-28T07:17:58.530 DEBUG:teuthology.orchestra.run.smithi156:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-06-28T07:17:58.531 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:17:59.537 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:00.542 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:01.547 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:02.552 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:03.557 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:04.563 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:05.569 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:06.575 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:07.580 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:08.240 WARNING:teuthology.provision.fog.smithi195:[Errno None] Unable to connect to port 22 on 172.21.15.195 2023-06-28T07:18:08.586 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:09.591 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:10.596 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:11.601 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:12.606 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:13.475 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None 2023-06-28T07:18:18.481 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:18:23.488 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:23.564 ERROR:teuthology.orchestra.connection:Error authenticating with smithi195.front.sepia.ceph.com: Authentication failed. 2023-06-28T07:18:23.565 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-06-28T07:18:23.565 DEBUG:teuthology.orchestra.run.smithi195:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-06-28T07:18:23.566 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:24.495 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:24.573 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:25.505 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:25.581 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:26.513 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:26.591 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:27.520 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:27.599 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:28.528 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:28.606 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:29.537 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:29.616 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:30.546 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:30.626 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:31.556 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:31.635 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:32.563 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:32.646 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:33.572 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:33.655 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:34.580 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:34.664 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:35.589 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:35.672 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:36.598 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:36.682 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:37.606 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:37.691 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:38.553 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None 2023-06-28T07:18:38.997 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:38.998 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:40.003 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:40.003 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:41.011 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:41.012 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:42.017 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:42.018 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:43.023 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:43.024 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:44.030 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:44.030 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:45.036 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:45.036 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:46.042 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:46.043 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:47.048 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:47.049 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:48.054 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:48.055 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:49.060 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:49.061 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:50.066 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:50.067 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:51.073 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:51.074 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:52.079 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:52.080 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:53.086 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:53.087 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:54.085 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None 2023-06-28T07:18:57.297 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:57.298 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:58.303 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:58.303 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:59.309 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:18:59.309 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:00.315 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:00.315 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:01.321 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:01.322 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:02.327 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:02.328 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:03.333 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:03.334 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:04.340 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:04.341 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:05.346 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:05.347 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:06.352 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:06.352 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:07.358 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:07.358 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:08.363 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:08.364 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:09.369 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:09.370 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:10.375 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:10.375 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:11.381 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:11.381 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:12.387 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:12.388 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:12.420 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None 2023-06-28T07:19:17.425 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:17.426 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:18.431 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:18.432 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:19.438 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:19.438 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:20.444 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:20.444 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:21.450 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:21.450 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:22.456 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:22.457 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:23.462 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:23.463 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:24.468 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:24.469 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:25.474 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:25.475 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:26.480 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:26.480 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:27.485 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:27.485 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:28.491 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:28.492 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:29.497 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:29.498 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:30.503 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:30.504 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:31.512 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:31.513 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:32.495 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None 2023-06-28T07:19:37.505 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:37.505 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:38.511 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:38.511 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:39.517 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:39.517 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:40.523 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:40.523 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:41.529 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:41.529 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:42.534 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:42.534 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:43.540 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:43.541 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:44.546 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:44.547 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:45.553 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:45.553 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:46.559 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:46.559 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:47.565 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:47.566 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:48.572 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:48.573 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:49.579 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:49.579 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:50.585 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:50.586 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:51.591 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:51.592 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:52.598 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:52.599 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:52.610 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None 2023-06-28T07:19:57.618 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:57.619 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:19:58.418 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 295, in _wait_for_ready self.remote.run(args=cmd, timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 97, in execute self.client.exec_command(self.command, timeout=self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 510, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 919, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1050, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2109, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 303, in read_all raise EOFError() EOFError 2023-06-28T07:19:58.419 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-28T07:19:58.420 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-28T07:20:03.430 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:20:03.508 INFO:teuthology.nuke:Checking targets against current locks 2023-06-28T07:20:08.517 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T07:20:12.162 DEBUG:teuthology.nuke:shortname: smithi150 2023-06-28T07:20:12.163 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T07:20:12.189 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-27_22:53:08-rados-reef-release-distro-default-smithi/7319103', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 07:14:25.615122', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-06-28T07:20:12.201 DEBUG:teuthology.nuke:shortname: smithi156 2023-06-28T07:20:12.202 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T07:20:12.229 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-27_22:53:08-rados-reef-release-distro-default-smithi/7319103', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 07:14:25.613669', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTmLPrv5YXCUEcH3IArOW9fhnaAouznAnWaIH6xGTXhaSHQAiMZBbbrklW6pQYRFfbU9BwDJwyVse/Nd3FflMxGttUKlM1KjRc5F+B/byny3KOk2VGL/ODyBFIIQgUAapRhsxEjOt7Jk2j6b3P/WKI0e6CiZfpgOry9Nqic/YLw/zpoqFSWHl0BIqlfitIKyPyyB6Kx5jN9yyC1hfI+3uwJ9pVsZFqdcC6YRzDGUVM5dzeum62rbZ/gO7tPzFtwmiHlcSNCTXd72AEv8ndY2+u9kyxHmgZM5rEaYTTTCrC4a7bST3rR3dMYC3rIrdwnjN2RxYEVBirUD2v/LZ1bZPzsW2sfYylMDKiVUm9r1LNZelZWuWb3E/74CJ89Y3Utt2m+hIXYO8VhbT9z3BYxcoJgsaLWsBbrpFCKNfFu1wvSEbPOfqF9rmYSzhhUrVaH/2uSjcB1lgcEtHDoSMsiOR7G6ggp2xmFhnFEaTKTahvmwQOmHC3qfCu9ZPYyIO2460='} 2023-06-28T07:20:12.242 DEBUG:teuthology.nuke:shortname: smithi195 2023-06-28T07:20:12.243 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T07:20:12.259 INFO:teuthology.orchestra.console:Power off smithi150 2023-06-28T07:20:12.260 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T07:20:12.272 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-27_22:53:08-rados-reef-release-distro-default-smithi/7319103', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 07:14:25.616611', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-06-28T07:20:12.282 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:20:12.300 INFO:teuthology.orchestra.console:Power off smithi156 2023-06-28T07:20:12.301 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T07:20:12.321 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:20:12.337 INFO:teuthology.orchestra.console:Power off smithi195 2023-06-28T07:20:12.338 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T07:20:12.360 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:20:12.716 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None 2023-06-28T07:20:17.726 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:20:17.735 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:20:17.746 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:20:17.820 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:20:21.848 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:20:21.858 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:20:22.837 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:20:26.960 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:20:26.968 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:20:27.178 INFO:teuthology.orchestra.console:Power off for smithi195 completed 2023-06-28T07:20:32.289 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T07:20:37.333 INFO:teuthology.orchestra.console:Power off for smithi156 completed 2023-06-28T07:20:37.334 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-06-28T07:20:37.359 INFO:teuthology.lock.ops:unlocked: smithi195.front.sepia.ceph.com 2023-06-28T07:20:42.441 INFO:teuthology.orchestra.console:Power off for smithi150 completed 2023-06-28T07:20:47.470 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:20:52.473 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-06-28T07:20:57.506 INFO:teuthology.lock.ops:unlocked: smithi156.front.sepia.ceph.com 2023-06-28T07:20:57.541 INFO:teuthology.lock.ops:unlocked: smithi150.front.sepia.ceph.com 2023-06-28T07:21:02.546 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T07:21:12.684 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-06-28T07:21:17.938 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 57, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 295, in _wait_for_ready self.remote.run(args=cmd, timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 97, in execute self.client.exec_command(self.command, timeout=self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 510, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 919, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1050, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2109, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 303, in read_all raise EOFError() EOFError