2023-06-28T16:09:32.129 INFO:root:teuthology version: 0.0.1.dev132+gaa9345a 2023-06-28T16:09:32.131 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-28T16:09:32.132 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-28T16:09:32.304 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f45938438b0>, signals=[15]) 2023-06-28T16:09:32.307 INFO:teuthology.lock.ops:Start node 'smithi071.front.sepia.ceph.com' reimaging 2023-06-28T16:09:32.308 INFO:teuthology.lock.ops:Updating [smithi071.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T16:09:32.308 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2023-06-28T16:09:32.333 INFO:teuthology.lock.ops:Start node 'smithi188.front.sepia.ceph.com' reimaging 2023-06-28T16:09:32.333 INFO:teuthology.lock.ops:Updating [smithi188.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T16:09:32.334 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2023-06-28T16:09:32.335 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /host {"name": "smithi071"} 2023-06-28T16:09:42.365 DEBUG:teuthology.provision.fog.smithi188:FOG request: GET /host {"name": "smithi188"} 2023-06-28T16:09:57.399 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /image {"name": "smithi_rhel_8.4"} 2023-06-28T16:10:07.436 DEBUG:teuthology.provision.fog.smithi188:FOG request: GET /image {"name": "smithi_rhel_8.4"} 2023-06-28T16:10:12.442 DEBUG:teuthology.provision.fog.smithi071:Requesting image smithi_rhel_8.4 (ID 7) 2023-06-28T16:10:12.442 DEBUG:teuthology.provision.fog.smithi071:FOG request: PUT /host/176 {"imageID": 7} 2023-06-28T16:10:17.496 DEBUG:teuthology.provision.fog.smithi188:Requesting image smithi_rhel_8.4 (ID 7) 2023-06-28T16:10:17.496 DEBUG:teuthology.provision.fog.smithi188:FOG request: PUT /host/293 {"imageID": 7} 2023-06-28T16:10:22.499 INFO:teuthology.provision.fog.smithi071:Scheduling deploy of rhel 8.4 2023-06-28T16:10:22.499 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /task/active None 2023-06-28T16:10:26.339 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T16:10:31.346 INFO:teuthology.provision.fog.smithi188:Scheduling deploy of rhel 8.4 2023-06-28T16:10:31.346 DEBUG:teuthology.provision.fog.smithi188:FOG request: GET /task/active None 2023-06-28T16:10:36.402 DEBUG:teuthology.provision.fog.smithi071:FOG request: POST /host/176/task {"taskTypeID": 1} 2023-06-28T16:10:36.411 DEBUG:teuthology.provision.fog.smithi188:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T16:10:36.453 DEBUG:teuthology.provision.fog.smithi188:FOG request: POST /host/293/task {"taskTypeID": 1} 2023-06-28T16:10:36.470 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /task/active None 2023-06-28T16:10:36.518 DEBUG:teuthology.provision.fog.smithi188:FOG request: GET /task/active None 2023-06-28T16:10:46.582 INFO:teuthology.orchestra.console:Power off smithi188 2023-06-28T16:10:46.583 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T16:10:46.598 INFO:teuthology.orchestra.console:Power off smithi071 2023-06-28T16:10:46.599 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T16:10:46.607 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:10:46.620 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:10:50.634 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:10:50.645 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:10:55.745 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:10:55.754 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:11:00.857 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:11:00.866 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:11:05.970 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:11:05.980 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:11:06.190 INFO:teuthology.orchestra.console:Power off for smithi071 completed 2023-06-28T16:11:06.291 INFO:teuthology.orchestra.console:Power on smithi071 2023-06-28T16:11:06.291 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T16:11:06.314 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:11:10.340 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:11:11.083 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:11:15.452 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:11:16.194 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:11:16.404 INFO:teuthology.orchestra.console:Power off for smithi188 completed 2023-06-28T16:11:16.506 INFO:teuthology.orchestra.console:Power on smithi188 2023-06-28T16:11:16.506 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T16:11:20.564 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:11:21.531 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:11:25.676 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:11:26.542 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:11:30.788 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:11:31.000 INFO:teuthology.orchestra.console:Power on for smithi071 completed 2023-06-28T16:11:31.101 INFO:teuthology.provision.fog.smithi071:Waiting for deploy to finish 2023-06-28T16:11:31.102 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /task/active None 2023-06-28T16:11:36.109 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:11:41.223 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:11:46.336 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:11:51.178 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /task/active None 2023-06-28T16:11:56.185 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:12:01.297 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:12:06.412 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:12:11.264 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /task/active None 2023-06-28T16:12:16.274 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:12:21.388 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:12:26.501 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:12:31.352 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /task/active None 2023-06-28T16:12:36.363 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:12:41.478 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:12:46.594 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:12:46.810 INFO:teuthology.orchestra.console:Power on for smithi188 completed 2023-06-28T16:12:46.911 INFO:teuthology.provision.fog.smithi188:Waiting for deploy to finish 2023-06-28T16:12:46.912 DEBUG:teuthology.provision.fog.smithi188:FOG request: GET /task/active None 2023-06-28T16:12:51.915 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /task/active None 2023-06-28T16:12:56.818 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:13:09.217 ERROR:teuthology.orchestra.connection:Error authenticating with smithi188.front.sepia.ceph.com: Authentication failed. 2023-06-28T16:13:09.219 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-06-28T16:13:09.220 DEBUG:teuthology.orchestra.run.smithi188:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-06-28T16:13:09.221 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:10.226 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:11.232 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:11.815 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /task/active None 2023-06-28T16:13:16.824 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:17.830 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:18.835 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:19.840 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:20.845 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:21.851 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:22.856 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:23.862 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:24.868 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:25.873 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:26.880 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:27.886 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:28.891 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:29.896 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:30.902 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:31.905 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /task/active None 2023-06-28T16:13:31.954 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:32.959 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:33.964 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:34.969 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:35.975 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:36.980 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:37.986 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:38.991 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:39.997 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:41.002 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:42.008 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:43.014 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:44.020 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:45.025 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:46.031 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:47.041 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /task/active None 2023-06-28T16:13:52.047 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:52.119 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:13:57.125 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:58.130 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:13:59.135 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:14:00.142 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:14:01.147 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:14:02.153 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:14:03.158 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:14:04.165 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:14:05.172 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:14:06.177 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:14:15.668 WARNING:teuthology.provision.fog.smithi071:[Errno None] Unable to connect to port 22 on 172.21.15.71 2023-06-28T16:14:21.670 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:14:29.744 WARNING:teuthology.provision.fog.smithi071:[Errno None] Unable to connect to port 22 on 172.21.15.71 2023-06-28T16:14:35.746 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:14:43.824 WARNING:teuthology.provision.fog.smithi071:[Errno None] Unable to connect to port 22 on 172.21.15.71 2023-06-28T16:14:49.826 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:14:52.880 WARNING:teuthology.provision.fog.smithi071:[Errno None] Unable to connect to port 22 on 172.21.15.71 2023-06-28T16:14:58.882 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:15:17.360 WARNING:teuthology.provision.fog.smithi071:[Errno None] Unable to connect to port 22 on 172.21.15.71 2023-06-28T16:15:23.362 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:15:26.416 WARNING:teuthology.provision.fog.smithi071:[Errno None] Unable to connect to port 22 on 172.21.15.71 2023-06-28T16:15:32.418 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:15:35.472 WARNING:teuthology.provision.fog.smithi071:[Errno None] Unable to connect to port 22 on 172.21.15.71 2023-06-28T16:15:41.474 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:15:41.876 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-06-28T16:15:41.876 DEBUG:teuthology.orchestra.run.smithi071:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-06-28T16:15:57.237 INFO:teuthology.provision.fog.smithi071:Node is ready 2023-06-28T16:15:57.238 DEBUG:teuthology.orchestra.run.smithi071:> hostname 2023-06-28T16:15:57.253 INFO:teuthology.orchestra.run.smithi071.stdout:smithi036 2023-06-28T16:15:57.253 DEBUG:teuthology.orchestra.run.smithi071:> grep smithi036 /etc/hosts 2023-06-28T16:15:57.308 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.36 smithi036.front.sepia.ceph.com smithi036 2023-06-28T16:15:57.309 DEBUG:teuthology.orchestra.run.smithi071:> sudo hostname smithi071 2023-06-28T16:15:57.474 DEBUG:teuthology.orchestra.run.smithi071:> sudo sed -i -e 's/smithi036/smithi071/g' /etc/hosts 2023-06-28T16:15:57.500 DEBUG:teuthology.orchestra.run.smithi071:> sudo sed -i -e 's/172.21.15.36/172.21.15.71/g' /etc/hosts 2023-06-28T16:15:57.564 DEBUG:teuthology.orchestra.run.smithi071:> sudo sed -i -e 's/smithi036/smithi071/g' /etc/hostname 2023-06-28T16:15:57.629 DEBUG:teuthology.orchestra.run.smithi071:> sudo hostname smithi071 2023-06-28T16:15:57.691 DEBUG:teuthology.orchestra.run.smithi071:> cat /etc/os-release 2023-06-28T16:15:57.745 INFO:teuthology.orchestra.run.smithi071.stdout:NAME="Red Hat Enterprise Linux" 2023-06-28T16:15:57.745 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION="8.4 (Ootpa)" 2023-06-28T16:15:57.745 INFO:teuthology.orchestra.run.smithi071.stdout:ID="rhel" 2023-06-28T16:15:57.746 INFO:teuthology.orchestra.run.smithi071.stdout:ID_LIKE="fedora" 2023-06-28T16:15:57.746 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_ID="8.4" 2023-06-28T16:15:57.746 INFO:teuthology.orchestra.run.smithi071.stdout:PLATFORM_ID="platform:el8" 2023-06-28T16:15:57.746 INFO:teuthology.orchestra.run.smithi071.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-06-28T16:15:57.746 INFO:teuthology.orchestra.run.smithi071.stdout:ANSI_COLOR="0;31" 2023-06-28T16:15:57.746 INFO:teuthology.orchestra.run.smithi071.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-06-28T16:15:57.747 INFO:teuthology.orchestra.run.smithi071.stdout:HOME_URL="https://www.redhat.com/" 2023-06-28T16:15:57.747 INFO:teuthology.orchestra.run.smithi071.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-06-28T16:15:57.747 INFO:teuthology.orchestra.run.smithi071.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-28T16:15:57.747 INFO:teuthology.orchestra.run.smithi071.stdout: 2023-06-28T16:15:57.747 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-28T16:15:57.747 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-06-28T16:15:57.748 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-06-28T16:15:57.748 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-06-28T16:15:57.749 INFO:teuthology.provision.fog.smithi071:Deploy complete! 2023-06-28T16:15:57.750 DEBUG:teuthology.parallel:result is None 2023-06-28T16:16:21.104 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2023-06-28T16:16:21.107 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 295, in _wait_for_ready self.remote.run(args=cmd, timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 97, in execute self.client.exec_command(self.command, timeout=self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 510, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 919, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1050, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2109, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) ConnectionResetError: [Errno 104] Connection reset by peer 2023-06-28T16:16:21.108 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-28T16:16:21.109 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-28T16:16:23.877 INFO:teuthology.nuke:Checking targets against current locks 2023-06-28T16:16:33.925 DEBUG:teuthology.nuke:shortname: smithi071 2023-06-28T16:16:33.926 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T16:16:38.962 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-28_14:13:04-rados-quincy_release-distro-default-smithi/7320327', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 16:09:31.243634', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMg4UFqEtb13ioLkR583XPy872giBLs9q+Lu4U8NslDEANlZtHyEUUWQ1xaaG5lRIhrOKuuxKNTeOEGQ4kBf4E4oCOOushxzIr6/omxblVwBGIBFDD3NBltaDde8SvT74Zs+dz/H6vt7wSVq4DZZHbENRFO6bDKH6vn8KnNylyesBJ5Ny5fVlwmaEalD+5j1i38CvixF0075VFOyN/mGvuXY8HdTJo5LVorv536gc3F2GDOCmM1s1Ndf2bAQxoBxYvC8q1/HkW/AXGr0U/Rt2YqcS00JZ7ZX97WbKB4ZXWzo5zTl+Z7QPVt0QQWD24yolnIhKKPbSEXc24P7wxvMxtVo/8b3eNEhSzYfTzBew7uChN0eqGfq5g0WvG7sUphlu8ScKfKYWrYzB3gEAwm7t2javGS89xsXg295s92bG65GkKJhLN+O4/SiNzyTVR1NerTE0LmV5AJYOGNABbYBtx5gZyOchGRfNgpsyi50nO71+HMyGixFjpT1z3NvZc9ZE='} 2023-06-28T16:16:38.969 DEBUG:teuthology.nuke:shortname: smithi188 2023-06-28T16:16:38.970 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T16:16:39.002 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-28_14:13:04-rados-quincy_release-distro-default-smithi/7320327', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 16:09:31.245527', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC45vpfgcvpz+ImvBUaRxVtlYAcWzCsjlx4DxfPstfKkraFTvLMSu//9b3rxBM3QPPBSoT6s7Nmypb0v/1rcFIkwgJlx1KExdC6JEG3SdPbGk+1STrbE1uQBSJ3c0fsKaSGbJbH0NHEYY1hEaG7BgIHwvGuN6gqTC8QnH1XZvhJoiJ8jmPE9NlxTxdxAihf1qVJ/Fpgt4kO1WKR6D0Vtx079To47jUVMqH3faneNp518cBcuyqCRDIwOHoLs6gn6L+Mj28kz1IY2IsR2tdyzhRiJ8p+5XduwwfA3pCc8M1+ONgSK5lq33flcaLc4fVDHwTtql4FJ9GxpJQq7Xb+OFLtp0ZzmVZ5Fktn//7bQSHuj2yxTfQw7OrJVHMQ47n68TMOlAFKz7m6xFtFUoxTWHYB01iAHcwrcQJkCJ4excGdr6FXGgzDBPxHki5EeTrJklCpQxPuMRyzwbOm/zZZNz5g2VLnHopa7za3MMx1IQCyFdJA80wT51ypMGkIudJdFwc='} 2023-06-28T16:16:39.017 INFO:teuthology.orchestra.console:Power off smithi071 2023-06-28T16:16:39.018 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T16:16:39.040 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:16:39.055 INFO:teuthology.orchestra.console:Power off smithi188 2023-06-28T16:16:39.056 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T16:16:39.076 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:16:43.068 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:16:43.098 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:16:47.182 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:16:47.210 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:16:47.395 INFO:teuthology.orchestra.console:Power off for smithi071 completed 2023-06-28T16:16:47.423 INFO:teuthology.orchestra.console:Power off for smithi188 completed 2023-06-28T16:16:57.570 INFO:teuthology.lock.ops:unlocked: smithi188.front.sepia.ceph.com 2023-06-28T16:16:57.571 INFO:teuthology.lock.ops:unlocked: smithi071.front.sepia.ceph.com 2023-06-28T16:17:02.810 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 57, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 295, in _wait_for_ready self.remote.run(args=cmd, timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 97, in execute self.client.exec_command(self.command, timeout=self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 510, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 919, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1050, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2109, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) ConnectionResetError: [Errno 104] Connection reset by peer