2023-06-28T16:22:05.298 INFO:root:teuthology version: 0.0.1.dev132+gaa9345a 2023-06-28T16:22:05.300 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-28T16:22:05.300 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-28T16:22:05.496 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7bcdfd6430>, signals=[15]) 2023-06-28T16:22:05.497 INFO:teuthology.lock.ops:Start node 'smithi057.front.sepia.ceph.com' reimaging 2023-06-28T16:22:05.498 INFO:teuthology.lock.ops:Updating [smithi057.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T16:22:05.498 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server 2023-06-28T16:22:05.526 INFO:teuthology.lock.ops:Start node 'smithi115.front.sepia.ceph.com' reimaging 2023-06-28T16:22:05.526 INFO:teuthology.lock.ops:Updating [smithi115.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T16:22:05.527 INFO:teuthology.lock.ops:Updating smithi115.front.sepia.ceph.com on lock server 2023-06-28T16:22:05.528 DEBUG:teuthology.provision.fog.smithi057:FOG request: GET /host {"name": "smithi057"} 2023-06-28T16:22:15.558 INFO:teuthology.lock.ops:Start node 'smithi171.front.sepia.ceph.com' reimaging 2023-06-28T16:22:15.558 INFO:teuthology.lock.ops:Updating [smithi171.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T16:22:15.558 INFO:teuthology.lock.ops:Updating smithi171.front.sepia.ceph.com on lock server 2023-06-28T16:22:15.559 DEBUG:teuthology.provision.fog.smithi115:FOG request: GET /host {"name": "smithi115"} 2023-06-28T16:22:28.235 DEBUG:teuthology.provision.fog.smithi171:FOG request: GET /host {"name": "smithi171"} 2023-06-28T16:22:38.243 DEBUG:teuthology.provision.fog.smithi057:FOG request: GET /image {"name": "smithi_centos_8.stream"} 2023-06-28T16:22:53.260 DEBUG:teuthology.provision.fog.smithi115:FOG request: GET /image {"name": "smithi_centos_8.stream"} 2023-06-28T16:22:58.266 DEBUG:teuthology.provision.fog.smithi057:Requesting image smithi_centos_8.stream (ID 3) 2023-06-28T16:22:58.266 DEBUG:teuthology.provision.fog.smithi057:FOG request: PUT /host/162 {"imageID": 3} 2023-06-28T16:23:03.275 DEBUG:teuthology.provision.fog.smithi171:FOG request: GET /image {"name": "smithi_centos_8.stream"} 2023-06-28T16:23:08.283 DEBUG:teuthology.provision.fog.smithi115:Requesting image smithi_centos_8.stream (ID 3) 2023-06-28T16:23:08.284 DEBUG:teuthology.provision.fog.smithi115:FOG request: PUT /host/220 {"imageID": 3} 2023-06-28T16:23:13.293 INFO:teuthology.provision.fog.smithi057:Scheduling deploy of centos 8.stream 2023-06-28T16:23:13.293 DEBUG:teuthology.provision.fog.smithi057:FOG request: GET /task/active None 2023-06-28T16:23:18.298 DEBUG:teuthology.provision.fog.smithi171:Requesting image smithi_centos_8.stream (ID 3) 2023-06-28T16:23:18.298 DEBUG:teuthology.provision.fog.smithi171:FOG request: PUT /host/276 {"imageID": 3} 2023-06-28T16:23:20.249 INFO:teuthology.provision.fog.smithi115:Scheduling deploy of centos 8.stream 2023-06-28T16:23:20.249 DEBUG:teuthology.provision.fog.smithi115:FOG request: GET /task/active None 2023-06-28T16:23:20.335 DEBUG:teuthology.provision.fog.smithi115:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T16:23:25.342 INFO:teuthology.provision.fog.smithi171:Scheduling deploy of centos 8.stream 2023-06-28T16:23:25.343 DEBUG:teuthology.provision.fog.smithi171:FOG request: GET /task/active None 2023-06-28T16:23:25.344 DEBUG:teuthology.provision.fog.smithi057:Canceling deploy task with ID 352236 2023-06-28T16:23:25.345 DEBUG:teuthology.provision.fog.smithi057:FOG request: DELETE /task/cancel {"id": 352236} 2023-06-28T16:23:35.391 DEBUG:teuthology.provision.fog.smithi115:FOG request: POST /host/220/task {"taskTypeID": 1} 2023-06-28T16:23:40.400 DEBUG:teuthology.provision.fog.smithi171:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T16:23:40.401 DEBUG:teuthology.provision.fog.smithi057:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T16:23:50.450 DEBUG:teuthology.provision.fog.smithi171:FOG request: POST /host/276/task {"taskTypeID": 1} 2023-06-28T16:23:55.458 DEBUG:teuthology.provision.fog.smithi115:FOG request: GET /task/active None 2023-06-28T16:23:55.459 DEBUG:teuthology.provision.fog.smithi057:FOG request: POST /host/162/task {"taskTypeID": 1} 2023-06-28T16:24:05.545 DEBUG:teuthology.provision.fog.smithi171:FOG request: GET /task/active None 2023-06-28T16:24:10.501 DEBUG:teuthology.provision.fog.smithi057:FOG request: GET /task/active None 2023-06-28T16:24:15.548 INFO:teuthology.orchestra.console:Power off smithi115 2023-06-28T16:24:15.549 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T16:24:25.601 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:24:25.631 INFO:teuthology.orchestra.console:Power off smithi057 2023-06-28T16:24:25.632 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T16:24:25.644 INFO:teuthology.orchestra.console:Power off smithi171 2023-06-28T16:24:25.645 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T16:24:30.613 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:24:30.656 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:24:30.666 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:24:35.668 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:24:35.678 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:24:35.723 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:24:35.880 INFO:teuthology.orchestra.console:Power off for smithi057 completed 2023-06-28T16:24:35.933 INFO:teuthology.orchestra.console:Power off for smithi115 completed 2023-06-28T16:24:35.981 INFO:teuthology.orchestra.console:Power on smithi057 2023-06-28T16:24:35.982 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T16:24:36.002 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:24:36.035 INFO:teuthology.orchestra.console:Power on smithi115 2023-06-28T16:24:36.036 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T16:24:36.056 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:24:40.024 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:24:40.080 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:24:40.291 INFO:teuthology.orchestra.console:Power on for smithi115 completed 2023-06-28T16:24:40.392 INFO:teuthology.provision.fog.smithi115:Waiting for deploy to finish 2023-06-28T16:24:40.393 DEBUG:teuthology.provision.fog.smithi115:FOG request: GET /task/active None 2023-06-28T16:24:40.555 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:24:40.788 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:24:43.636 WARNING:teuthology.provision.fog.smithi115:[Errno None] Unable to connect to port 22 on 172.21.15.115 2023-06-28T16:24:44.135 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:24:44.426 INFO:teuthology.orchestra.console:Power on for smithi057 completed 2023-06-28T16:24:44.527 INFO:teuthology.provision.fog.smithi057:Waiting for deploy to finish 2023-06-28T16:24:44.528 DEBUG:teuthology.provision.fog.smithi057:FOG request: GET /task/active None 2023-06-28T16:24:45.819 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:24:45.900 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:24:48.880 WARNING:teuthology.provision.fog.smithi057:[Errno None] Unable to connect to port 22 on 172.21.15.57 2023-06-28T16:24:49.640 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:24:51.014 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:24:52.720 WARNING:teuthology.provision.fog.smithi115:[Errno None] Unable to connect to port 22 on 172.21.15.115 2023-06-28T16:24:54.885 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:24:59.892 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:24:59.903 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:25:04.909 WARNING:teuthology.provision.fog.smithi057:[Errno None] Unable to connect to port 22 on 172.21.15.57 2023-06-28T16:25:05.019 INFO:teuthology.orchestra.console:Power off for smithi171 completed 2023-06-28T16:25:05.119 INFO:teuthology.orchestra.console:Power on smithi171 2023-06-28T16:25:05.120 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T16:25:07.984 WARNING:teuthology.provision.fog.smithi115:[Errno None] Unable to connect to port 22 on 172.21.15.115 2023-06-28T16:25:10.913 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:25:15.917 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:25:15.920 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:25:18.993 WARNING:teuthology.provision.fog.smithi057:[Errno None] Unable to connect to port 22 on 172.21.15.57 2023-06-28T16:25:18.993 WARNING:teuthology.provision.fog.smithi115:[Errno None] Unable to connect to port 22 on 172.21.15.115 2023-06-28T16:25:20.932 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:25:24.997 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:25:30.001 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:25:30.003 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:25:33.072 WARNING:teuthology.provision.fog.smithi115:[Errno None] Unable to connect to port 22 on 172.21.15.115 2023-06-28T16:25:33.073 WARNING:teuthology.provision.fog.smithi057:[Errno None] Unable to connect to port 22 on 172.21.15.57 2023-06-28T16:25:35.116 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:25:35.325 INFO:teuthology.orchestra.console:Power on for smithi171 completed 2023-06-28T16:25:35.426 INFO:teuthology.provision.fog.smithi171:Waiting for deploy to finish 2023-06-28T16:25:35.427 DEBUG:teuthology.provision.fog.smithi171:FOG request: GET /task/active None 2023-06-28T16:25:35.504 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:25:40.509 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:25:45.515 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:25:50.522 WARNING:teuthology.provision.fog.smithi115:[Errno None] Unable to connect to port 22 on 172.21.15.115 2023-06-28T16:25:50.523 WARNING:teuthology.provision.fog.smithi171:[Errno None] Unable to connect to port 22 on 172.21.15.171 2023-06-28T16:25:56.528 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:26:01.533 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:26:09.616 WARNING:teuthology.provision.fog.smithi171:[Errno None] Unable to connect to port 22 on 172.21.15.171 2023-06-28T16:26:15.623 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:26:18.704 WARNING:teuthology.provision.fog.smithi171:[Errno None] Unable to connect to port 22 on 172.21.15.171 2023-06-28T16:26:19.888 WARNING:teuthology.provision.fog.smithi115:[Errno None] Unable to connect to port 22 on 172.21.15.115 2023-06-28T16:26:22.056 ERROR:teuthology.orchestra.connection:Error authenticating with smithi057.front.sepia.ceph.com: Authentication failed. 2023-06-28T16:26:22.058 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-06-28T16:26:22.058 DEBUG:teuthology.orchestra.run.smithi057:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-06-28T16:26:22.059 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:23.065 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:24.069 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:24.706 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:26:25.074 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:25.889 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:26:30.896 WARNING:teuthology.provision.fog.smithi171:[Errno None] Unable to connect to port 22 on 172.21.15.171 2023-06-28T16:26:30.897 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:30.967 ERROR:teuthology.orchestra.connection:Error authenticating with smithi115.front.sepia.ceph.com: Authentication failed. 2023-06-28T16:26:30.967 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-06-28T16:26:30.967 DEBUG:teuthology.orchestra.run.smithi115:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-06-28T16:26:30.968 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:31.905 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:31.975 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:32.913 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:32.983 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:33.921 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:33.991 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:34.929 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:34.999 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:35.937 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:36.007 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:36.897 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:26:41.902 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:41.902 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:42.908 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:42.908 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:43.914 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:43.915 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:44.921 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:44.922 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:45.928 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:45.929 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:46.935 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:46.936 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:47.942 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:47.943 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:48.948 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:48.949 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:49.956 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:49.956 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:50.961 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:50.962 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:51.967 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:51.968 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:52.974 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:52.974 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:53.980 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:53.980 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:54.986 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:54.986 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:55.992 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:55.993 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:56.999 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:56.999 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:58.005 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:58.006 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:59.012 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:26:59.012 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:00.018 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:00.019 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:00.340 WARNING:teuthology.provision.fog.smithi171:[Errno None] Unable to connect to port 22 on 172.21.15.171 2023-06-28T16:27:01.025 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:01.025 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:02.031 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:02.032 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:03.038 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:03.038 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:04.046 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:04.046 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:05.052 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:05.053 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:06.059 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:06.059 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:06.342 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:27:06.459 ERROR:teuthology.orchestra.connection:Error authenticating with smithi171.front.sepia.ceph.com: Authentication failed. 2023-06-28T16:27:06.460 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-06-28T16:27:06.460 DEBUG:teuthology.orchestra.run.smithi171:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-06-28T16:27:06.461 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:07.066 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:07.067 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:07.469 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:08.073 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:08.073 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:08.477 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:09.080 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:09.081 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:09.484 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:10.089 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:10.089 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:10.493 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:11.099 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:11.099 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:11.502 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:12.107 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:12.108 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:12.511 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:13.117 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:13.117 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:13.520 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:14.125 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:14.125 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:14.528 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:15.133 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:15.134 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:15.538 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:16.142 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:16.143 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:16.545 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:17.151 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:17.151 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:17.554 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:18.159 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:18.160 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:18.563 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:19.168 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:19.169 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:19.572 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:20.177 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:20.177 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:20.581 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:21.186 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:21.186 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:21.590 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:22.195 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:22.598 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:23.204 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:23.607 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:24.212 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:24.615 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:25.624 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:26.633 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:27.643 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:28.652 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:29.662 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:30.671 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:31.681 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:32.690 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:33.700 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:34.709 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:35.718 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:36.728 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:37.737 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:38.745 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:39.754 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:40.764 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:41.773 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:42.782 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:43.792 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:44.801 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:45.809 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:46.818 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:47.828 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:48.837 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:49.848 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:50.857 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:51.867 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:52.876 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:53.884 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:54.893 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:55.903 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:56.912 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:57.921 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:58.931 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:27:59.941 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:28:00.951 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:28:01.961 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:28:02.973 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:28:03.982 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:28:04.992 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:28:06.004 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:28:07.014 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:29:35.728 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2023-06-28T16:29:35.731 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 295, in _wait_for_ready self.remote.run(args=cmd, timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 97, in execute self.client.exec_command(self.command, timeout=self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 510, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 919, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1050, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2109, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) ConnectionResetError: [Errno 104] Connection reset by peer 2023-06-28T16:29:35.732 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-28T16:29:35.733 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-28T16:29:36.182 INFO:teuthology.nuke:Checking targets against current locks 2023-06-28T16:29:36.240 DEBUG:teuthology.nuke:shortname: smithi057 2023-06-28T16:29:36.241 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T16:29:36.266 DEBUG:teuthology.nuke:shortname: smithi115 2023-06-28T16:29:36.267 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T16:29:36.269 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi057.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-28_14:13:04-rados-quincy_release-distro-default-smithi/7320358', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 16:22:03.834950', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-06-28T16:29:36.286 DEBUG:teuthology.nuke:shortname: smithi171 2023-06-28T16:29:36.287 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T16:29:36.290 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi115.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-28_14:13:04-rados-quincy_release-distro-default-smithi/7320358', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 16:22:03.833526', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpnJPiJ3+uxdCVksVkbmEQqqMDBE8tZGOy2/MZUx49nVgXC6GG6OGOFzoG0hoefJGnsbrQ4QziJOgZesQU5kNZerL2DDXeV52TzKxvVIApwsPUkj7vu94pdx7opODeM6w7EzRAoxVgLiP79KVpIJJZVYdIX4T46YgvZA0+g3DQCsxllbpt7zW7tpfSmtoTqhfuEg5/0B02wO/9IUZdApQY6qSE8k8wZl3uFwKARPVAh4npkPf7rNRsV8/qcednU8K+HjNLaCyDWwCOaBAcMN4RJ43q3LSuZ/xB2jL3dlYy51/23TZeC9FJznA79fuYl6RQDClHmec8GxL/Wg/F1jKXIR5lQSZrVjbEdfKUUBpl8tDArc5uP5kT3cjEE+ZkIikL5fYMwti3ZZl2XETjr6sbd8zli2pp8b63HMpk61tGNWv69+RbFuAPOx4ERPBYbJCG+N1qzZln+mNKfSCNlgpP8ru1V2f0RsIVS37ysLvNPcSl15Di1brUnv3oU+XcUxM='} 2023-06-28T16:29:36.311 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi171.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-28_14:13:04-rados-quincy_release-distro-default-smithi/7320358', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 16:22:03.836116', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFZFQE/8dtdEqmPdoqu7nZ7Q8V/RwD7YlvCtuBdW93Wb/G01hKkc1NWxPAq4CcRZbAzz2KOxMOXWafWo7dT/71PE5kK98J8oKtM7yLPO4sm+W4GuoR+WTqtzrFTvgTyxGQTsMGrXFcwe/C+4MX+qNBvEVo28onEmuQufX5/Xphzr9d/YAaLQcYoQCHPVbpE3v5bP2xXr9vYByJeda6b/Y15Adlmp97oEhPl2xl71xwSIu7yqgO03fT4OTHPbWdZWS3rzrFQ8vcOFIrRarXALmTxsgMN6BbsmaK5GzMF5o006otoYaiCXiaq4a1lvZFT+DqzNIU9QePTRCvd6ghi1qS9j6JMsFpQESg40lsHuo2pRkS7TOSSsA337kSsYs9WrZw9iodXpDsvxiU4a5qdVtuHkXRKid41U6ulHnBQaePgMisA4iJcLiI4XoNDr1IDGT/l7E3iA6I5ltKNBHsI8i/Ar2gmMBsEy/WTJSzZsxqGWrkIEQVWxfP0hhZefXxb58='} 2023-06-28T16:29:36.322 INFO:teuthology.orchestra.console:Power off smithi057 2023-06-28T16:29:36.323 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T16:29:36.342 INFO:teuthology.orchestra.console:Power off smithi115 2023-06-28T16:29:36.342 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T16:29:36.352 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:29:36.363 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:29:36.380 INFO:teuthology.orchestra.console:Power off smithi171 2023-06-28T16:29:36.380 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T16:29:36.402 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:29:39.838 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2023-06-28T16:29:40.390 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:29:40.398 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:29:40.427 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:29:45.500 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:29:45.509 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:29:45.541 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:29:50.610 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:29:50.621 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:29:50.653 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:29:55.722 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:29:55.732 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:29:55.764 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:30:00.834 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:30:00.844 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:30:00.875 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:30:05.946 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:30:05.955 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:30:05.986 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:30:11.058 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:30:11.068 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:30:11.098 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:30:11.278 INFO:teuthology.orchestra.console:Power off for smithi057 completed 2023-06-28T16:30:21.031 INFO:teuthology.orchestra.console:Power off for smithi171 completed 2023-06-28T16:30:21.031 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:30:21.066 INFO:teuthology.lock.ops:unlocked: smithi057.front.sepia.ceph.com 2023-06-28T16:30:23.071 INFO:teuthology.lock.ops:unlocked: smithi171.front.sepia.ceph.com 2023-06-28T16:30:27.999 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:30:33.113 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:30:38.224 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:30:43.335 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:30:48.447 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:30:53.558 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:30:54.669 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds 2023-06-28T16:30:54.670 ERROR:teuthology.orchestra.console:Failed to power off smithi115 2023-06-28T16:30:59.738 INFO:teuthology.lock.ops:unlocked: smithi115.front.sepia.ceph.com 2023-06-28T16:31:00.104 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 57, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 295, in _wait_for_ready self.remote.run(args=cmd, timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 97, in execute self.client.exec_command(self.command, timeout=self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 510, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 919, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1050, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2109, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) ConnectionResetError: [Errno 104] Connection reset by peer