2023-06-28T10:04:32.365 INFO:root:teuthology version: 0.0.1.dev132+gaa9345a 2023-06-28T10:04:32.366 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-28T10:04:32.367 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-28T10:04:32.571 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc0013074c0>, signals=[15]) 2023-06-28T10:04:32.573 INFO:teuthology.lock.ops:Start node 'smithi033.front.sepia.ceph.com' reimaging 2023-06-28T10:04:32.573 INFO:teuthology.lock.ops:Updating [smithi033.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T10:04:32.574 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server 2023-06-28T10:04:32.599 INFO:teuthology.lock.ops:Start node 'smithi110.front.sepia.ceph.com' reimaging 2023-06-28T10:04:32.600 INFO:teuthology.lock.ops:Updating [smithi110.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T10:04:32.600 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2023-06-28T10:04:32.602 DEBUG:teuthology.provision.fog.smithi033:FOG request: GET /host {"name": "smithi033"} 2023-06-28T10:04:34.051 INFO:teuthology.lock.ops:Start node 'smithi184.front.sepia.ceph.com' reimaging 2023-06-28T10:04:34.051 INFO:teuthology.lock.ops:Updating [smithi184.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T10:04:34.052 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2023-06-28T10:04:34.053 DEBUG:teuthology.provision.fog.smithi110:FOG request: GET /host {"name": "smithi110"} 2023-06-28T10:04:54.082 DEBUG:teuthology.provision.fog.smithi184:FOG request: GET /host {"name": "smithi184"} 2023-06-28T10:04:59.125 DEBUG:teuthology.provision.fog.smithi033:FOG request: GET /image {"name": "smithi_centos_8.stream"} 2023-06-28T10:05:04.095 DEBUG:teuthology.provision.fog.smithi110:FOG request: GET /image {"name": "smithi_centos_8.stream"} 2023-06-28T10:05:14.135 DEBUG:teuthology.provision.fog.smithi184:FOG request: GET /image {"name": "smithi_centos_8.stream"} 2023-06-28T10:05:19.143 DEBUG:teuthology.provision.fog.smithi110:Requesting image smithi_centos_8.stream (ID 3) 2023-06-28T10:05:19.143 DEBUG:teuthology.provision.fog.smithi110:FOG request: PUT /host/215 {"imageID": 3} 2023-06-28T10:05:19.145 DEBUG:teuthology.provision.fog.smithi033:Requesting image smithi_centos_8.stream (ID 3) 2023-06-28T10:05:19.146 DEBUG:teuthology.provision.fog.smithi033:FOG request: PUT /host/138 {"imageID": 3} 2023-06-28T10:05:29.197 DEBUG:teuthology.provision.fog.smithi184:Requesting image smithi_centos_8.stream (ID 3) 2023-06-28T10:05:29.197 DEBUG:teuthology.provision.fog.smithi184:FOG request: PUT /host/289 {"imageID": 3} 2023-06-28T10:05:34.205 INFO:teuthology.provision.fog.smithi110:Scheduling deploy of centos 8.stream 2023-06-28T10:05:34.205 DEBUG:teuthology.provision.fog.smithi110:FOG request: GET /task/active None 2023-06-28T10:05:34.206 INFO:teuthology.provision.fog.smithi033:Scheduling deploy of centos 8.stream 2023-06-28T10:05:34.207 DEBUG:teuthology.provision.fog.smithi033:FOG request: GET /task/active None 2023-06-28T10:05:44.259 INFO:teuthology.provision.fog.smithi184:Scheduling deploy of centos 8.stream 2023-06-28T10:05:44.260 DEBUG:teuthology.provision.fog.smithi184:FOG request: GET /task/active None 2023-06-28T10:05:49.267 DEBUG:teuthology.provision.fog.smithi033:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T10:05:49.269 DEBUG:teuthology.provision.fog.smithi110:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T10:05:59.319 DEBUG:teuthology.provision.fog.smithi110:FOG request: POST /host/215/task {"taskTypeID": 1} 2023-06-28T10:06:04.168 DEBUG:teuthology.provision.fog.smithi184:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T10:06:04.169 DEBUG:teuthology.provision.fog.smithi033:FOG request: POST /host/138/task {"taskTypeID": 1} 2023-06-28T10:06:14.216 DEBUG:teuthology.provision.fog.smithi184:FOG request: POST /host/289/task {"taskTypeID": 1} 2023-06-28T10:06:19.226 DEBUG:teuthology.provision.fog.smithi033:FOG request: GET /task/active None 2023-06-28T10:06:19.228 DEBUG:teuthology.provision.fog.smithi110:FOG request: GET /task/active None 2023-06-28T10:06:30.999 DEBUG:teuthology.provision.fog.smithi184:FOG request: GET /task/active None 2023-06-28T10:06:41.044 INFO:teuthology.orchestra.console:Power off smithi033 2023-06-28T10:06:41.045 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T10:06:41.059 INFO:teuthology.orchestra.console:Power off smithi110 2023-06-28T10:06:41.060 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T10:06:41.068 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:06:41.080 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:06:46.129 INFO:teuthology.orchestra.console:Power off smithi184 2023-06-28T10:06:46.130 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T10:06:46.197 INFO:teuthology.orchestra.console:Power off for smithi033 completed 2023-06-28T10:06:46.198 INFO:teuthology.orchestra.console:Power off for smithi110 completed 2023-06-28T10:06:46.299 INFO:teuthology.orchestra.console:Power on smithi033 2023-06-28T10:06:46.299 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T10:06:46.307 INFO:teuthology.orchestra.console:Power on smithi110 2023-06-28T10:06:46.308 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T10:06:51.155 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:06:51.323 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:06:51.333 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:06:55.360 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:06:56.167 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:06:56.333 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:06:56.379 INFO:teuthology.orchestra.console:Power off for smithi184 completed 2023-06-28T10:06:56.480 INFO:teuthology.orchestra.console:Power on smithi184 2023-06-28T10:06:56.481 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T10:06:56.505 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:07:00.473 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:07:00.531 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:07:01.446 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:07:04.644 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:07:05.584 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:07:06.559 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:07:09.756 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:07:10.696 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:07:10.906 INFO:teuthology.orchestra.console:Power on for smithi110 completed 2023-06-28T10:07:11.007 INFO:teuthology.provision.fog.smithi110:Waiting for deploy to finish 2023-06-28T10:07:11.008 DEBUG:teuthology.provision.fog.smithi110:FOG request: GET /task/active None 2023-06-28T10:07:16.013 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:07:16.021 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:07:16.107 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T10:07:16.224 INFO:teuthology.orchestra.console:Power on for smithi033 completed 2023-06-28T10:07:16.233 INFO:teuthology.orchestra.console:Power on for smithi184 completed 2023-06-28T10:07:16.325 INFO:teuthology.provision.fog.smithi033:Waiting for deploy to finish 2023-06-28T10:07:16.325 DEBUG:teuthology.provision.fog.smithi033:FOG request: GET /task/active None 2023-06-28T10:07:21.330 INFO:teuthology.provision.fog.smithi184:Waiting for deploy to finish 2023-06-28T10:07:21.330 DEBUG:teuthology.provision.fog.smithi184:FOG request: GET /task/active None 2023-06-28T10:07:21.331 WARNING:teuthology.provision.fog.smithi110:[Errno None] Unable to connect to port 22 on 172.21.15.110 2023-06-28T10:07:24.476 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T10:07:24.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T10:07:27.336 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T10:07:35.412 WARNING:teuthology.provision.fog.smithi110:[Errno None] Unable to connect to port 22 on 172.21.15.110 2023-06-28T10:07:41.416 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T10:07:46.421 WARNING:teuthology.provision.fog.smithi184:[Errno None] Unable to connect to port 22 on 172.21.15.184 2023-06-28T10:07:46.422 WARNING:teuthology.provision.fog.smithi033:[Errno None] Unable to connect to port 22 on 172.21.15.33 2023-06-28T10:07:49.492 WARNING:teuthology.provision.fog.smithi110:[Errno None] Unable to connect to port 22 on 172.21.15.110 2023-06-28T10:07:52.424 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T10:07:52.427 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T10:07:55.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T10:08:00.503 WARNING:teuthology.provision.fog.smithi184:[Errno None] Unable to connect to port 22 on 172.21.15.184 2023-06-28T10:08:03.568 WARNING:teuthology.provision.fog.smithi110:[Errno None] Unable to connect to port 22 on 172.21.15.110 2023-06-28T10:08:06.504 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T10:08:11.510 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T10:08:16.516 WARNING:teuthology.provision.fog.smithi033:[Errno None] Unable to connect to port 22 on 172.21.15.33 2023-06-28T10:08:19.568 WARNING:teuthology.provision.fog.smithi110:[Errno None] Unable to connect to port 22 on 172.21.15.110 2023-06-28T10:08:22.520 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T10:08:22.638 ERROR:teuthology.orchestra.connection:Error authenticating with smithi033.front.sepia.ceph.com: Authentication failed. 2023-06-28T10:08:22.640 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-06-28T10:08:22.641 DEBUG:teuthology.orchestra.run.smithi033:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-06-28T10:08:22.642 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:23.646 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:24.652 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:25.570 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T10:08:25.659 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:26.665 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:27.677 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:28.628 WARNING:teuthology.provision.fog.smithi110:[Errno None] Unable to connect to port 22 on 172.21.15.110 2023-06-28T10:08:28.682 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:29.688 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:30.064 WARNING:teuthology.provision.fog.smithi184:[Errno None] Unable to connect to port 22 on 172.21.15.184 2023-06-28T10:08:30.693 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:31.699 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:32.704 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:33.710 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:34.629 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T10:08:39.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T10:08:44.638 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:44.708 ERROR:teuthology.orchestra.connection:Error authenticating with smithi110.front.sepia.ceph.com: Authentication failed. 2023-06-28T10:08:44.709 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-06-28T10:08:44.710 DEBUG:teuthology.orchestra.run.smithi110:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-06-28T10:08:44.710 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:44.748 ERROR:teuthology.orchestra.connection:Error authenticating with smithi184.front.sepia.ceph.com: Authentication failed. 2023-06-28T10:08:44.749 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-06-28T10:08:44.749 DEBUG:teuthology.orchestra.run.smithi184:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-06-28T10:08:44.750 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:45.646 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:45.719 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:45.758 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:46.656 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:46.728 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:46.768 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:47.667 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:47.738 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:47.777 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:48.677 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:48.748 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:48.786 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:49.686 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:49.758 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:49.795 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:50.697 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:50.767 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:50.805 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:51.708 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:51.777 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:51.816 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:52.716 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:52.785 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:52.824 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:53.727 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:53.793 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:53.831 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:54.736 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:54.801 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:54.841 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:55.747 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:55.811 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:55.852 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:56.756 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:56.820 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:56.859 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:57.767 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:57.830 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:57.869 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:58.777 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:58.841 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:58.878 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:59.787 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:59.851 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:08:59.888 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:00.798 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:00.862 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:00.897 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:01.809 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:01.873 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:01.907 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:02.819 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:02.883 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:02.917 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:03.830 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:03.894 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:03.928 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:04.840 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:04.904 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:04.938 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:05.849 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:05.913 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:05.949 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:06.859 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:06.924 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:06.959 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:07.869 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:07.934 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:07.969 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:08.879 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:08.943 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:08.980 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:09.889 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:09.953 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:09.990 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:10.899 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:10.963 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:10.999 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:11.909 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:11.973 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:12.010 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:12.918 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:12.983 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:13.019 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:13.932 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:13.992 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:14.033 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:14.942 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:15.002 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:15.043 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:15.952 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:16.012 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:16.052 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:16.962 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:17.022 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:17.063 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:18.033 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:18.073 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:19.042 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:19.083 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:20.052 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:20.093 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:21.062 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:21.102 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:22.073 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:22.113 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:23.081 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:23.123 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:24.092 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:24.134 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:25.101 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:25.143 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:26.110 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:26.152 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:27.121 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:27.163 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:28.130 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:28.173 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:29.140 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:29.183 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:30.151 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:30.193 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:31.161 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:31.202 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:32.172 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:32.211 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:33.219 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:34.228 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:35.238 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:36.246 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:09:37.255 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T10:11:05.456 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2023-06-28T10:11:05.459 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 295, in _wait_for_ready self.remote.run(args=cmd, timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 97, in execute self.client.exec_command(self.command, timeout=self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 510, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 919, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1050, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2109, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) ConnectionResetError: [Errno 104] Connection reset by peer 2023-06-28T10:11:05.460 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-28T10:11:05.461 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-28T10:11:05.544 INFO:teuthology.nuke:Checking targets against current locks 2023-06-28T10:11:05.608 DEBUG:teuthology.nuke:shortname: smithi033 2023-06-28T10:11:05.609 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T10:11:05.639 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-27_22:57:36-orch-reef-release-distro-default-smithi/7319501', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 10:04:31.523797', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-06-28T10:11:05.642 DEBUG:teuthology.nuke:shortname: smithi110 2023-06-28T10:11:05.643 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T10:11:05.657 DEBUG:teuthology.nuke:shortname: smithi184 2023-06-28T10:11:05.658 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T10:11:05.670 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-27_22:57:36-orch-reef-release-distro-default-smithi/7319501', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 10:04:31.525180', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGb/pJDO6FJIZtwj+Myl6frtd3HuALzRqVs6odS+6QQEC0d2lFYAkwkJLkB/RjstvSFwzf4GLpeyonSSj8bm9PwudsfLhYavSwwtg7BIM7qqzHLxUgUFGuD4Cmu0fEtAHg+pF9F6nwizDE/valf0KHuI7XGadXVTNQCCelaykKXvH8HmKBGjYfEmouLAcHPhRd+C9YKhZCX6/UWEIb/beFAsR/yXgKfnNjIwyweJbtkEQN8Lb33O4nTF2tz6uCkxV8nO6nKCV4e5z3A7xLliPMFBen1Uyi8n3jhnyNq2L204zOb2QKYdrGfveySMcObvR1qy+yJDwr1EHDSqEPIXIdpQLplPLtzSJZeHmLW4py6Sro/FgQO30QTZxsxo2UjlZd+oIR5WIO79l+wVfg28gsKYNxggsx4G1bvNtuSqI+RU5gRQaJgSwR5lBa0IQhrdrZoBgwmNKSp86YgERdviln3lL4SubX0PAxbL2lTI4404Rs70Ixi1Pyk2w0TkgGd0M='} 2023-06-28T10:11:05.678 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-27_22:57:36-orch-reef-release-distro-default-smithi/7319501', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 10:04:31.522224', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-06-28T10:11:05.702 INFO:teuthology.orchestra.console:Power off smithi033 2023-06-28T10:11:05.703 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T10:11:05.725 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:11:05.741 INFO:teuthology.orchestra.console:Power off smithi110 2023-06-28T10:11:05.742 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T10:11:05.755 INFO:teuthology.orchestra.console:Power off smithi184 2023-06-28T10:11:05.756 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T10:11:09.766 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:11:10.785 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:11:13.877 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:11:15.794 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:11:17.991 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:11:18.201 INFO:teuthology.orchestra.console:Power off for smithi033 completed 2023-06-28T10:11:23.309 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:11:23.318 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2023-06-28T10:11:23.321 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:11:25.642 INFO:teuthology.lock.ops:unlocked: smithi033.front.sepia.ceph.com 2023-06-28T10:11:25.678 INFO:teuthology.orchestra.console:Power off for smithi184 completed 2023-06-28T10:11:25.833 INFO:teuthology.lock.ops:unlocked: smithi184.front.sepia.ceph.com 2023-06-28T10:11:27.346 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:11:31.459 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:11:36.569 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:11:41.679 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:11:46.792 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:11:51.904 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:11:57.017 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:12:02.129 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:12:07.240 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:12:12.354 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:12:17.465 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:12:22.576 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:12:27.686 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:12:32.797 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:12:33.910 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds 2023-06-28T10:12:33.911 ERROR:teuthology.orchestra.console:Failed to power off smithi110 2023-06-28T10:12:43.979 INFO:teuthology.lock.ops:unlocked: smithi110.front.sepia.ceph.com 2023-06-28T10:12:59.390 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 57, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 295, in _wait_for_ready self.remote.run(args=cmd, timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 97, in execute self.client.exec_command(self.command, timeout=self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 510, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 919, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1050, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2109, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) ConnectionResetError: [Errno 104] Connection reset by peer