2023-06-28T11:00:58.623 INFO:root:teuthology version: 0.0.1.dev132+gaa9345a 2023-06-28T11:00:58.624 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-28T11:00:58.625 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-28T11:01:18.585 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f970e06a670>, signals=[15]) 2023-06-28T11:01:18.588 INFO:teuthology.lock.ops:Start node 'smithi089.front.sepia.ceph.com' reimaging 2023-06-28T11:01:18.589 INFO:teuthology.lock.ops:Updating [smithi089.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T11:01:18.589 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2023-06-28T11:01:18.610 INFO:teuthology.lock.ops:Start node 'smithi133.front.sepia.ceph.com' reimaging 2023-06-28T11:01:18.610 INFO:teuthology.lock.ops:Updating [smithi133.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T11:01:18.610 INFO:teuthology.lock.ops:Updating smithi133.front.sepia.ceph.com on lock server 2023-06-28T11:01:18.612 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /host {"name": "smithi089"} 2023-06-28T11:01:23.643 INFO:teuthology.lock.ops:Start node 'smithi190.front.sepia.ceph.com' reimaging 2023-06-28T11:01:23.643 INFO:teuthology.lock.ops:Updating [smithi190.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T11:01:23.644 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2023-06-28T11:01:23.645 DEBUG:teuthology.provision.fog.smithi133:FOG request: GET /host {"name": "smithi133"} 2023-06-28T11:01:34.969 DEBUG:teuthology.provision.fog.smithi190:FOG request: GET /host {"name": "smithi190"} 2023-06-28T11:01:40.669 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /image {"name": "smithi_rhel_8.6"} 2023-06-28T11:01:50.684 DEBUG:teuthology.provision.fog.smithi133:FOG request: GET /image {"name": "smithi_rhel_8.6"} 2023-06-28T11:01:55.690 DEBUG:teuthology.provision.fog.smithi089:Requesting image smithi_rhel_8.6 (ID 5) 2023-06-28T11:01:55.691 DEBUG:teuthology.provision.fog.smithi089:FOG request: PUT /host/194 {"imageID": 5} 2023-06-28T11:01:55.749 DEBUG:teuthology.provision.fog.smithi190:FOG request: GET /image {"name": "smithi_rhel_8.6"} 2023-06-28T11:01:55.830 DEBUG:teuthology.provision.fog.smithi133:Requesting image smithi_rhel_8.6 (ID 5) 2023-06-28T11:01:55.831 DEBUG:teuthology.provision.fog.smithi133:FOG request: PUT /host/238 {"imageID": 5} 2023-06-28T11:02:00.841 DEBUG:teuthology.provision.fog.smithi190:Requesting image smithi_rhel_8.6 (ID 5) 2023-06-28T11:02:00.841 DEBUG:teuthology.provision.fog.smithi190:FOG request: PUT /host/295 {"imageID": 5} 2023-06-28T11:02:00.842 INFO:teuthology.provision.fog.smithi089:Scheduling deploy of rhel 8.6 2023-06-28T11:02:00.842 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None 2023-06-28T11:02:10.905 INFO:teuthology.provision.fog.smithi190:Scheduling deploy of rhel 8.6 2023-06-28T11:02:10.906 DEBUG:teuthology.provision.fog.smithi190:FOG request: GET /task/active None 2023-06-28T11:02:15.917 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T11:02:15.919 INFO:teuthology.provision.fog.smithi133:Scheduling deploy of rhel 8.6 2023-06-28T11:02:15.919 DEBUG:teuthology.provision.fog.smithi133:FOG request: GET /task/active None 2023-06-28T11:02:25.978 DEBUG:teuthology.provision.fog.smithi089:FOG request: POST /host/194/task {"taskTypeID": 1} 2023-06-28T11:02:26.054 DEBUG:teuthology.provision.fog.smithi190:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T11:02:26.065 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None 2023-06-28T11:02:26.093 DEBUG:teuthology.provision.fog.smithi133:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T11:02:26.098 DEBUG:teuthology.provision.fog.smithi190:FOG request: POST /host/295/task {"taskTypeID": 1} 2023-06-28T11:02:26.136 DEBUG:teuthology.provision.fog.smithi133:FOG request: POST /host/238/task {"taskTypeID": 1} 2023-06-28T11:02:31.142 DEBUG:teuthology.provision.fog.smithi190:FOG request: GET /task/active None 2023-06-28T11:02:36.269 INFO:teuthology.orchestra.console:Power off smithi089 2023-06-28T11:02:36.270 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T11:02:36.340 DEBUG:teuthology.provision.fog.smithi133:FOG request: GET /task/active None 2023-06-28T11:02:46.353 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:02:46.394 INFO:teuthology.orchestra.console:Power off smithi190 2023-06-28T11:02:46.394 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T11:02:51.437 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:02:51.467 INFO:teuthology.orchestra.console:Power off smithi133 2023-06-28T11:02:51.468 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T11:02:51.579 INFO:teuthology.orchestra.console:Power off for smithi190 completed 2023-06-28T11:02:51.681 INFO:teuthology.orchestra.console:Power on smithi190 2023-06-28T11:02:51.681 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T11:02:51.707 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:02:55.440 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:02:55.732 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:02:55.942 INFO:teuthology.orchestra.console:Power on for smithi190 completed 2023-06-28T11:02:56.043 INFO:teuthology.provision.fog.smithi190:Waiting for deploy to finish 2023-06-28T11:02:56.044 DEBUG:teuthology.provision.fog.smithi190:FOG request: GET /task/active None 2023-06-28T11:02:56.120 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T11:03:00.552 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:03:01.500 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:03:05.664 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:03:06.512 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:03:10.776 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:03:11.625 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:03:14.544 WARNING:teuthology.provision.fog.smithi190:[Errno None] Unable to connect to port 22 on 172.21.15.190 2023-06-28T11:03:15.888 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:03:16.736 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:03:20.550 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T11:03:25.553 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:03:25.563 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:03:26.106 INFO:teuthology.orchestra.console:Power off for smithi089 completed 2023-06-28T11:03:26.207 INFO:teuthology.orchestra.console:Power on smithi089 2023-06-28T11:03:26.208 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T11:03:26.227 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:03:28.624 WARNING:teuthology.provision.fog.smithi190:[Errno None] Unable to connect to port 22 on 172.21.15.190 2023-06-28T11:03:30.248 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:03:30.674 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:03:34.360 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:03:34.570 INFO:teuthology.orchestra.console:Power on for smithi089 completed 2023-06-28T11:03:34.627 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T11:03:39.628 INFO:teuthology.provision.fog.smithi089:Waiting for deploy to finish 2023-06-28T11:03:39.629 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None 2023-06-28T11:03:39.631 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:03:42.024 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T11:03:42.026 INFO:teuthology.orchestra.console:Power off for smithi133 completed 2023-06-28T11:03:42.127 INFO:teuthology.orchestra.console:Power on smithi133 2023-06-28T11:03:42.127 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T11:03:42.153 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:03:42.704 WARNING:teuthology.provision.fog.smithi190:[Errno None] Unable to connect to port 22 on 172.21.15.190 2023-06-28T11:03:45.104 WARNING:teuthology.provision.fog.smithi089:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-06-28T11:03:46.175 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:03:48.708 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T11:03:53.713 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T11:03:58.716 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:03:58.726 WARNING:teuthology.provision.fog.smithi190:[Errno None] Unable to connect to port 22 on 172.21.15.190 2023-06-28T11:04:01.776 WARNING:teuthology.provision.fog.smithi089:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-06-28T11:04:03.829 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:04:04.729 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T11:04:09.733 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T11:04:09.836 INFO:teuthology.orchestra.console:Power on for smithi133 completed 2023-06-28T11:04:09.937 INFO:teuthology.provision.fog.smithi133:Waiting for deploy to finish 2023-06-28T11:04:09.937 DEBUG:teuthology.provision.fog.smithi133:FOG request: GET /task/active None 2023-06-28T11:04:14.944 WARNING:teuthology.provision.fog.smithi089:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-06-28T11:04:15.032 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T11:04:20.946 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T11:04:23.088 WARNING:teuthology.provision.fog.smithi133:[Errno None] Unable to connect to port 22 on 172.21.15.133 2023-06-28T11:04:24.016 WARNING:teuthology.provision.fog.smithi089:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-06-28T11:04:28.276 WARNING:teuthology.provision.fog.smithi190:[Errno None] Unable to connect to port 22 on 172.21.15.190 2023-06-28T11:04:29.091 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T11:04:34.097 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T11:04:39.101 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T11:04:44.105 WARNING:teuthology.provision.fog.smithi133:[Errno None] Unable to connect to port 22 on 172.21.15.133 2023-06-28T11:04:44.181 ERROR:teuthology.orchestra.connection:Error authenticating with smithi190.front.sepia.ceph.com: Authentication failed. 2023-06-28T11:04:44.182 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-06-28T11:04:44.184 DEBUG:teuthology.orchestra.run.smithi190:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-06-28T11:04:44.184 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:04:45.190 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:04:46.193 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:04:47.197 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:04:48.201 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:04:49.205 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:04:50.107 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T11:04:55.113 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:04:56.117 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:04:57.121 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:04:57.456 WARNING:teuthology.provision.fog.smithi089:[Errno None] Unable to connect to port 22 on 172.21.15.89 2023-06-28T11:04:58.125 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:04:58.192 WARNING:teuthology.provision.fog.smithi133:[Errno None] Unable to connect to port 22 on 172.21.15.133 2023-06-28T11:04:59.129 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:00.135 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:01.140 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:02.146 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:03.151 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:03.458 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T11:05:08.465 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T11:05:13.470 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:13.540 ERROR:teuthology.orchestra.connection:Error authenticating with smithi089.front.sepia.ceph.com: Authentication failed. 2023-06-28T11:05:13.541 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-06-28T11:05:13.541 DEBUG:teuthology.orchestra.run.smithi089:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-06-28T11:05:13.542 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:14.480 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:14.550 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:15.489 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:15.559 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:16.498 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:16.528 WARNING:teuthology.provision.fog.smithi133:[Errno None] Unable to connect to port 22 on 172.21.15.133 2023-06-28T11:05:16.568 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:17.507 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:17.577 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:18.515 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:18.586 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:19.526 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:19.596 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:20.535 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:20.605 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:21.543 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:21.613 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:22.530 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T11:05:22.552 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:22.623 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:23.562 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:23.632 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:24.571 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:24.646 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:25.581 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:25.584 WARNING:teuthology.provision.fog.smithi133:[Errno None] Unable to connect to port 22 on 172.21.15.133 2023-06-28T11:05:25.655 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:26.591 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:26.663 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:27.601 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:27.673 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:28.610 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:28.683 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:29.619 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:29.691 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:30.628 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:30.701 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:31.586 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T11:05:36.594 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:36.594 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:36.661 ERROR:teuthology.orchestra.connection:Error authenticating with smithi133.front.sepia.ceph.com: Authentication failed. 2023-06-28T11:05:36.662 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-06-28T11:05:36.663 DEBUG:teuthology.orchestra.run.smithi133:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-06-28T11:05:36.664 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:37.602 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:37.673 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:38.611 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:38.683 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:39.621 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:39.693 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:40.631 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:40.703 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:41.640 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:41.714 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:42.648 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:42.724 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:43.658 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:43.733 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:44.667 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:44.742 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:45.676 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:45.751 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:46.685 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:46.761 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:47.695 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:47.771 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:48.705 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:48.781 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:49.716 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:49.791 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:50.725 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:50.802 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:51.735 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:51.812 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:52.746 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:52.822 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:53.754 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:53.833 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:54.765 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:54.843 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:55.774 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:55.854 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:56.784 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:56.867 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:57.793 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:57.877 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:58.803 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:58.886 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:59.814 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:05:59.895 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:00.823 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:00.904 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:01.835 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:01.913 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:02.846 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:02.922 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:03.856 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:03.932 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:04.866 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:04.939 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:05.877 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:05.949 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:06.961 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:07.970 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:08.981 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:09.992 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:11.002 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:12.011 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:13.021 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:14.027 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:15.038 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:16.049 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:17.058 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:18.070 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:19.080 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:20.092 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:21.102 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:22.112 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:23.121 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:24.131 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:06:25.139 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T11:07:51.280 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2023-06-28T11:07:51.285 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 295, in _wait_for_ready self.remote.run(args=cmd, timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 97, in execute self.client.exec_command(self.command, timeout=self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 510, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 919, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1050, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2109, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) ConnectionResetError: [Errno 104] Connection reset by peer 2023-06-28T11:07:51.287 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-28T11:07:51.287 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-28T11:07:51.374 INFO:teuthology.nuke:Checking targets against current locks 2023-06-28T11:07:51.424 DEBUG:teuthology.nuke:shortname: smithi089 2023-06-28T11:07:51.424 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T11:07:51.451 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-27_22:58:43-fs-reef-release-distro-default-smithi/7319604', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 11:00:37.769079', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0='} 2023-06-28T11:07:51.456 DEBUG:teuthology.nuke:shortname: smithi133 2023-06-28T11:07:51.457 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T11:07:51.478 DEBUG:teuthology.nuke:shortname: smithi190 2023-06-28T11:07:51.478 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T11:07:51.482 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi133.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-27_22:58:43-fs-reef-release-distro-default-smithi/7319604', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 11:00:37.777829', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCukyTzGNIRqTLHO0MohGr3wzKCGOkw4kJS5c5EeuaMOt+3GM/dC80ZTlaMI9hYpyVU1BVdwFqjb1JOTydSysRInK3hQuw4dwGuWb7QtgrUKLPc7Uf0Ka7QkBLNUgEbJlfXB8W4vYmtu4OaTDOgnT1nEO/n1ixs36tfS87l4c/LuYa6qodFlv5bWx4ESTLvl9sQSxNJ01MxJBtl8jKbpo75ZLC64JKEbZYEu9okKCnOYfXeJLsixwD9XjZ50UF3ov4sPeKoMa6WeXRWzvhBz0lpccJK09Kdbkno6ooj2jYy4hvFgCUznUPChdFyoa1vBo0QTz+RIg2na5nyzOG/0/BFjMi3h8OkST1ckSo1pI4+/QVyLorELrnIJW0OeW7t5AEnvCpeYGBDfuPIRG8bagrVIJI2/4VsGVdlDxDXRclq+PHEguFxXiF2RCNIUJP2HckN20bmdUSwSPBs/q4My5/DJBYj/1oHWa/k3U8wxSmQ9CpkR6tHpM3SlPQmvbgLnhE='} 2023-06-28T11:07:51.499 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-27_22:58:43-fs-reef-release-distro-default-smithi/7319604', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 11:00:37.773688', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-06-28T11:07:51.508 INFO:teuthology.orchestra.console:Power off smithi089 2023-06-28T11:07:51.509 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T11:07:51.532 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:07:51.556 INFO:teuthology.orchestra.console:Power off smithi133 2023-06-28T11:07:51.556 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T11:07:51.572 INFO:teuthology.orchestra.console:Power off smithi190 2023-06-28T11:07:51.573 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T11:07:51.583 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:07:51.597 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:07:55.583 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:07:55.613 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:07:55.624 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:08:00.695 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:08:00.727 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:08:00.737 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:08:00.909 INFO:teuthology.orchestra.console:Power off for smithi089 completed 2023-06-28T11:08:07.257 INFO:teuthology.orchestra.console:Power off for smithi133 completed 2023-06-28T11:08:07.286 INFO:teuthology.lock.ops:unlocked: smithi089.front.sepia.ceph.com 2023-06-28T11:08:07.412 INFO:teuthology.lock.ops:unlocked: smithi133.front.sepia.ceph.com 2023-06-28T11:08:10.018 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:08:15.130 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:08:20.241 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:08:25.351 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T11:08:25.564 INFO:teuthology.orchestra.console:Power off for smithi190 completed 2023-06-28T11:08:33.949 INFO:teuthology.lock.ops:unlocked: smithi190.front.sepia.ceph.com 2023-06-28T11:08:34.282 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 57, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 295, in _wait_for_ready self.remote.run(args=cmd, timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 97, in execute self.client.exec_command(self.command, timeout=self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 510, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 919, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1050, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2109, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) ConnectionResetError: [Errno 104] Connection reset by peer