2023-06-28T09:51:06.913 INFO:root:teuthology version: 0.0.1.dev132+gaa9345a 2023-06-28T09:51:06.914 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-28T09:51:06.915 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-28T09:51:07.066 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe1d7b073a0>, signals=[15]) 2023-06-28T09:51:07.068 INFO:teuthology.lock.ops:Start node 'smithi008.front.sepia.ceph.com' reimaging 2023-06-28T09:51:07.068 INFO:teuthology.lock.ops:Updating [smithi008.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T09:51:07.069 INFO:teuthology.lock.ops:Updating smithi008.front.sepia.ceph.com on lock server 2023-06-28T09:51:07.094 INFO:teuthology.lock.ops:Start node 'smithi130.front.sepia.ceph.com' reimaging 2023-06-28T09:51:07.095 INFO:teuthology.lock.ops:Updating [smithi130.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T09:51:07.095 INFO:teuthology.lock.ops:Updating smithi130.front.sepia.ceph.com on lock server 2023-06-28T09:51:07.097 DEBUG:teuthology.provision.fog.smithi008:FOG request: GET /host {"name": "smithi008"} 2023-06-28T09:51:17.133 DEBUG:teuthology.provision.fog.smithi130:FOG request: GET /host {"name": "smithi130"} 2023-06-28T09:51:25.019 DEBUG:teuthology.provision.fog.smithi008:FOG request: GET /image {"name": "smithi_centos_8.stream"} 2023-06-28T09:51:30.052 DEBUG:teuthology.provision.fog.smithi130:FOG request: GET /image {"name": "smithi_centos_8.stream"} 2023-06-28T09:51:32.748 DEBUG:teuthology.provision.fog.smithi008:Requesting image smithi_centos_8.stream (ID 3) 2023-06-28T09:51:32.749 DEBUG:teuthology.provision.fog.smithi008:FOG request: PUT /host/114 {"imageID": 3} 2023-06-28T09:51:32.809 DEBUG:teuthology.provision.fog.smithi130:Requesting image smithi_centos_8.stream (ID 3) 2023-06-28T09:51:32.809 DEBUG:teuthology.provision.fog.smithi130:FOG request: PUT /host/235 {"imageID": 3} 2023-06-28T09:51:32.837 INFO:teuthology.provision.fog.smithi008:Scheduling deploy of centos 8.stream 2023-06-28T09:51:32.838 DEBUG:teuthology.provision.fog.smithi008:FOG request: GET /task/active None 2023-06-28T09:51:32.878 INFO:teuthology.provision.fog.smithi130:Scheduling deploy of centos 8.stream 2023-06-28T09:51:32.879 DEBUG:teuthology.provision.fog.smithi130:FOG request: GET /task/active None 2023-06-28T09:51:32.924 DEBUG:teuthology.provision.fog.smithi008:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T09:51:32.963 DEBUG:teuthology.provision.fog.smithi130:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T09:51:32.966 DEBUG:teuthology.provision.fog.smithi008:FOG request: POST /host/114/task {"taskTypeID": 1} 2023-06-28T09:51:33.006 DEBUG:teuthology.provision.fog.smithi130:FOG request: POST /host/235/task {"taskTypeID": 1} 2023-06-28T09:51:33.046 DEBUG:teuthology.provision.fog.smithi008:FOG request: GET /task/active None 2023-06-28T09:51:33.074 DEBUG:teuthology.provision.fog.smithi130:FOG request: GET /task/active None 2023-06-28T09:51:48.127 INFO:teuthology.orchestra.console:Power off smithi130 2023-06-28T09:51:48.128 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T09:51:48.142 INFO:teuthology.orchestra.console:Power off smithi008 2023-06-28T09:51:48.143 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T09:51:48.151 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T09:51:48.163 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T09:51:48.273 INFO:teuthology.orchestra.console:Power off for smithi130 completed 2023-06-28T09:51:48.284 INFO:teuthology.orchestra.console:Power off for smithi008 completed 2023-06-28T09:51:48.374 INFO:teuthology.orchestra.console:Power on smithi130 2023-06-28T09:51:48.375 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T09:51:48.386 INFO:teuthology.orchestra.console:Power on smithi008 2023-06-28T09:51:48.386 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T09:51:48.394 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T09:51:48.406 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T09:51:52.420 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T09:51:52.429 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T09:51:52.640 INFO:teuthology.orchestra.console:Power on for smithi008 completed 2023-06-28T09:51:52.741 INFO:teuthology.provision.fog.smithi008:Waiting for deploy to finish 2023-06-28T09:51:52.741 DEBUG:teuthology.provision.fog.smithi008:FOG request: GET /task/active None 2023-06-28T09:51:52.823 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T09:51:56.536 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T09:51:56.746 INFO:teuthology.orchestra.console:Power on for smithi130 completed 2023-06-28T09:51:56.847 INFO:teuthology.provision.fog.smithi130:Waiting for deploy to finish 2023-06-28T09:51:56.847 DEBUG:teuthology.provision.fog.smithi130:FOG request: GET /task/active None 2023-06-28T09:52:00.272 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T09:52:11.376 WARNING:teuthology.provision.fog.smithi008:[Errno None] Unable to connect to port 22 on 172.21.15.8 2023-06-28T09:52:17.380 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T09:52:22.386 WARNING:teuthology.provision.fog.smithi130:[Errno None] Unable to connect to port 22 on 172.21.15.130 2023-06-28T09:52:25.456 WARNING:teuthology.provision.fog.smithi008:[Errno None] Unable to connect to port 22 on 172.21.15.8 2023-06-28T09:52:28.390 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T09:52:33.397 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T09:52:38.399 WARNING:teuthology.provision.fog.smithi130:[Errno None] Unable to connect to port 22 on 172.21.15.130 2023-06-28T09:52:41.456 WARNING:teuthology.provision.fog.smithi008:[Errno None] Unable to connect to port 22 on 172.21.15.8 2023-06-28T09:52:44.401 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T09:52:47.461 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T09:52:48.451 WARNING:teuthology.provision.fog.smithi130:[Errno None] Unable to connect to port 22 on 172.21.15.130 2023-06-28T09:52:51.508 WARNING:teuthology.provision.fog.smithi008:[Errno None] Unable to connect to port 22 on 172.21.15.8 2023-06-28T09:52:54.455 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T09:53:04.465 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T09:53:22.800 WARNING:teuthology.provision.fog.smithi130:[Errno None] Unable to connect to port 22 on 172.21.15.130 2023-06-28T09:53:24.961 ERROR:teuthology.orchestra.connection:Error authenticating with smithi008.front.sepia.ceph.com: Authentication failed. 2023-06-28T09:53:24.962 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-06-28T09:53:24.963 DEBUG:teuthology.orchestra.run.smithi008:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-06-28T09:53:24.964 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:25.969 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:26.975 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:27.979 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:28.801 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T09:53:33.808 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:33.883 ERROR:teuthology.orchestra.connection:Error authenticating with smithi130.front.sepia.ceph.com: Authentication failed. 2023-06-28T09:53:33.884 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-06-28T09:53:33.884 DEBUG:teuthology.orchestra.run.smithi130:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-06-28T09:53:33.885 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:34.816 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:34.892 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:35.823 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:35.899 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:36.831 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:36.907 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:37.840 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:37.915 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:38.848 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:38.923 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:39.856 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:39.931 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:40.864 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:40.939 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:41.873 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:41.948 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:42.881 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:42.957 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:43.890 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:43.966 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:44.899 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:44.974 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:45.909 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:45.984 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:46.918 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:46.994 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:47.927 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:48.002 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:48.935 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:49.010 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:49.943 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:50.018 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:50.951 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:51.026 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:51.960 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:52.035 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:52.969 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:53.045 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:53.978 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:54.053 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:54.986 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:55.062 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:55.995 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:56.071 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:57.004 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:57.079 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:58.012 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:58.087 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:59.019 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:53:59.095 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:00.028 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:00.103 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:01.037 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:01.112 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:02.046 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:02.120 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:03.055 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:03.128 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:04.063 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:04.137 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:05.071 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:05.145 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:06.080 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:06.153 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:07.089 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:07.162 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:08.098 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:08.172 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:09.107 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:09.181 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:10.116 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:10.189 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:11.125 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:11.198 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:12.133 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:12.206 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:13.142 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:13.215 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:14.150 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:14.225 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:15.159 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:15.234 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:16.168 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:16.243 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:17.177 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:17.251 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:18.186 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:18.260 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:19.195 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:19.268 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:20.204 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:20.277 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:21.212 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:21.286 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:22.221 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:22.295 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:23.305 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:24.315 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:25.324 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:54:26.334 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T09:56:37.104 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2023-06-28T09:56:37.109 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 295, in _wait_for_ready self.remote.run(args=cmd, timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 97, in execute self.client.exec_command(self.command, timeout=self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 510, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 919, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1050, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2109, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) ConnectionResetError: [Errno 104] Connection reset by peer 2023-06-28T09:56:37.110 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-28T09:56:37.110 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-28T09:56:39.984 INFO:teuthology.nuke:Checking targets against current locks 2023-06-28T09:56:40.032 DEBUG:teuthology.nuke:shortname: smithi008 2023-06-28T09:56:40.033 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T09:56:40.052 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi008.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-27_22:58:43-fs-reef-release-distro-default-smithi/7319474', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 09:51:06.102214', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-06-28T09:56:40.055 DEBUG:teuthology.nuke:shortname: smithi130 2023-06-28T09:56:40.055 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T09:56:40.075 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi130.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-27_22:58:43-fs-reef-release-distro-default-smithi/7319474', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 09:51:06.103502', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDyYghJZuz+rx7jQv5/HOVc5W7Y+YKjvs64hhqGqlVfTQEmGXX4OjNX13/E8qM9Bgssp4E07fb2xCcVZNHQ/FiO6GN7kraZHknRHexpN4awJoRDPB737ZayzmpPStOK46eqwzzaKRNFaqClXusDossd0CMGiI/HI45muO1ompDMAlLcDbx2DYHjgTfQe9xDWKuFEHTITUZfYIH+7lhM5CXNh+/Psa1zexFQDDlNW1W+pZOm8AAAdfzrIdaP8Y338aGzRmnUeJc5uvMOplSY1SgYkCYE+u4Vu1tp0g41zSLc7r+CZbxNsyHo4Z0Tm4Jt0/PxN50hGWx21RtkNuzfM/f3slGNcRtFM8lCCdfnPorcj1IUnn8T3JzqByf/3bY+RoGidO2bP+z2wkMGpDyltwORfYvE9lTX3e8LzTlD3XcZXw0lDlO90xj7gV3SgG5xN6gb1u1LOn1eNBScfRc/ZQTK/td04NjIhYq4HfR7ndGAI49TJWgUglk3PpRhVmgDkhs='} 2023-06-28T09:56:40.105 INFO:teuthology.orchestra.console:Power off smithi008 2023-06-28T09:56:40.106 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T09:56:40.128 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T09:56:40.146 INFO:teuthology.orchestra.console:Power off smithi130 2023-06-28T09:56:40.147 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T09:56:40.169 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T09:56:41.200 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2023-06-28T09:56:44.160 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T09:56:44.194 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T09:56:48.274 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T09:56:48.305 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T09:56:48.485 INFO:teuthology.orchestra.console:Power off for smithi008 completed 2023-06-28T09:56:48.518 INFO:teuthology.orchestra.console:Power off for smithi130 completed 2023-06-28T09:56:55.305 INFO:teuthology.lock.ops:unlocked: smithi130.front.sepia.ceph.com 2023-06-28T09:56:55.312 INFO:teuthology.lock.ops:unlocked: smithi008.front.sepia.ceph.com 2023-06-28T09:56:55.580 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 57, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 295, in _wait_for_ready self.remote.run(args=cmd, timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 97, in execute self.client.exec_command(self.command, timeout=self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 510, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 919, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1050, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2109, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) ConnectionResetError: [Errno 104] Connection reset by peer