2024-04-18T05:28:40.896 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-18T05:28:40.896 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-18T05:28:41.039 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7473a82c10>, signals=[15]) 2024-04-18T05:28:41.039 INFO:teuthology.lock.ops:Start node 'smithi144.front.sepia.ceph.com' reimaging 2024-04-18T05:28:41.039 INFO:teuthology.lock.ops:Updating [smithi144.front.sepia.ceph.com]: reset os type and version on server 2024-04-18T05:28:41.040 INFO:teuthology.lock.ops:Updating smithi144.front.sepia.ceph.com on lock server 2024-04-18T05:28:41.062 INFO:teuthology.lock.ops:Start node 'smithi186.front.sepia.ceph.com' reimaging 2024-04-18T05:28:41.062 INFO:teuthology.lock.ops:Updating [smithi186.front.sepia.ceph.com]: reset os type and version on server 2024-04-18T05:28:41.062 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2024-04-18T05:28:41.181 DEBUG:teuthology.provision.fog.smithi144:Requesting image smithi_rhel_8.6 (ID 5) 2024-04-18T05:28:41.209 DEBUG:teuthology.provision.fog.smithi186:Requesting image smithi_rhel_8.6 (ID 5) 2024-04-18T05:28:41.221 INFO:teuthology.provision.fog.smithi144:Scheduling deploy of rhel 8.6 2024-04-18T05:28:41.260 INFO:teuthology.provision.fog.smithi186:Scheduling deploy of rhel 8.6 2024-04-18T05:28:41.433 INFO:teuthology.orchestra.console.smithi144:Power off 2024-04-18T05:28:41.433 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-18T05:28:41.456 DEBUG:teuthology.orchestra.console.smithi144:power off output: Chassis Power Control: Down/Off 2024-04-18T05:28:41.456 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T05:28:41.473 INFO:teuthology.orchestra.console.smithi186:Power off 2024-04-18T05:28:41.473 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-18T05:28:41.481 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is off 2024-04-18T05:28:41.492 DEBUG:teuthology.orchestra.console.smithi186:power off output: Chassis Power Control: Down/Off 2024-04-18T05:28:41.492 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T05:28:41.512 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is off 2024-04-18T05:28:41.582 INFO:teuthology.orchestra.console.smithi144:Power off completed 2024-04-18T05:28:41.613 INFO:teuthology.orchestra.console.smithi186:Power off completed 2024-04-18T05:28:41.683 INFO:teuthology.orchestra.console.smithi144:Power on 2024-04-18T05:28:41.684 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-04-18T05:28:41.705 DEBUG:teuthology.orchestra.console.smithi144:power on output: Chassis Power Control: Up/On 2024-04-18T05:28:41.705 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T05:28:41.714 INFO:teuthology.orchestra.console.smithi186:Power on 2024-04-18T05:28:41.714 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-04-18T05:28:41.728 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is off 2024-04-18T05:28:41.735 DEBUG:teuthology.orchestra.console.smithi186:power on output: Chassis Power Control: Up/On 2024-04-18T05:28:41.735 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T05:28:41.758 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is off 2024-04-18T05:28:45.732 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T05:28:45.759 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T05:28:45.842 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is off 2024-04-18T05:28:45.870 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is off 2024-04-18T05:28:49.846 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T05:28:49.871 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T05:28:49.955 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is on 2024-04-18T05:28:49.980 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-04-18T05:28:50.057 INFO:teuthology.orchestra.console.smithi144:Power on completed 2024-04-18T05:28:50.081 INFO:teuthology.orchestra.console.smithi186:Power on completed 2024-04-18T05:28:50.158 INFO:teuthology.provision.fog.smithi144:Waiting for deploy to finish 2024-04-18T05:28:50.182 INFO:teuthology.provision.fog.smithi186:Waiting for deploy to finish 2024-04-18T05:30:05.590 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:30:08.543 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-04-18T05:30:14.547 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:30:14.686 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-04-18T05:30:20.688 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:30:20.830 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-04-18T05:30:26.835 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:30:26.974 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-04-18T05:30:32.979 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:30:33.122 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-04-18T05:30:39.127 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:30:39.266 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-04-18T05:30:45.271 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:30:45.410 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-04-18T05:30:51.413 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:30:51.554 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-04-18T05:30:57.561 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:30:57.694 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-04-18T05:31:03.699 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:31:03.838 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-04-18T05:31:09.843 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:31:09.986 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-04-18T05:31:15.991 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:31:16.126 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-04-18T05:31:22.129 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:31:22.274 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-04-18T05:31:28.281 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:31:28.414 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-04-18T05:31:34.420 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:31:34.558 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-04-18T05:31:36.008 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:31:39.070 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-04-18T05:31:40.562 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:31:40.702 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-04-18T05:31:45.075 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:31:46.706 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:31:48.126 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-04-18T05:31:54.135 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:31:57.214 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-04-18T05:32:03.221 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:32:05.022 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-04-18T05:32:06.302 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-04-18T05:32:11.028 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:32:12.304 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:32:12.885 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-04-18T05:32:15.358 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-04-18T05:32:18.888 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:32:19.020 ERROR:teuthology.orchestra.connection:Error authenticating with smithi144.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2024-04-18T05:32:21.360 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:32:23.170 ERROR:teuthology.orchestra.connection:Error authenticating with smithi144.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2024-04-18T05:32:24.421 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-04-18T05:32:30.256 ERROR:teuthology.orchestra.connection:Error authenticating with smithi144.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2024-04-18T05:32:30.423 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:32:33.506 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-04-18T05:32:39.508 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:32:40.380 ERROR:teuthology.orchestra.connection:Error authenticating with smithi144.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2024-04-18T05:32:42.558 WARNING:teuthology.provision.fog.smithi186:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-04-18T05:32:48.560 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:32:53.471 ERROR:teuthology.orchestra.connection:Error authenticating with smithi144.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2024-04-18T05:33:09.474 ERROR:paramiko.transport:Exception (client): Error reading SSH protocol banner 2024-04-18T05:33:09.476 ERROR:paramiko.transport:Traceback (most recent call last): 2024-04-18T05:33:09.476 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2327, in _check_banner 2024-04-18T05:33:09.476 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2024-04-18T05:33:09.476 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 381, in readline 2024-04-18T05:33:09.476 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2024-04-18T05:33:09.476 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 618, in _read_timeout 2024-04-18T05:33:09.476 ERROR:paramiko.transport: raise EOFError() 2024-04-18T05:33:09.476 ERROR:paramiko.transport:EOFError 2024-04-18T05:33:09.476 ERROR:paramiko.transport: 2024-04-18T05:33:09.476 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2024-04-18T05:33:09.476 ERROR:paramiko.transport: 2024-04-18T05:33:09.476 ERROR:paramiko.transport:Traceback (most recent call last): 2024-04-18T05:33:09.476 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2143, in run 2024-04-18T05:33:09.477 ERROR:paramiko.transport: self._check_banner() 2024-04-18T05:33:09.477 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2331, in _check_banner 2024-04-18T05:33:09.477 ERROR:paramiko.transport: raise SSHException( 2024-04-18T05:33:09.477 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2024-04-18T05:33:09.477 ERROR:paramiko.transport: 2024-04-18T05:33:09.477 ERROR:teuthology.orchestra.connection:Error authenticating with smithi144.front.sepia.ceph.com: SSHException('Error reading SSH protocol banner') (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2327, in _check_banner buf = self.packetizer.readline(timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 381, in readline buf += self._read_timeout(timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 618, in _read_timeout raise EOFError() EOFError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 451, in connect t.start_client(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 738, in start_client raise e File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2143, in run self._check_banner() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2331, in _check_banner raise SSHException( paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2024-04-18T05:33:48.563 WARNING:teuthology.provision.fog.smithi186:timed out 2024-04-18T05:33:54.566 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:33:54.800 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-04-18T05:33:54.801 DEBUG:teuthology.orchestra.run.smithi186:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-04-18T05:33:55.066 INFO:teuthology.provision.fog.smithi186:Node is ready 2024-04-18T05:33:55.067 DEBUG:teuthology.orchestra.run.smithi186:> hostname 2024-04-18T05:33:55.084 INFO:teuthology.orchestra.run.smithi186.stdout:smithi186.front.sepia.ceph.com 2024-04-18T05:33:55.084 DEBUG:teuthology.orchestra.run.smithi186:> grep smithi186.front.sepia.ceph.com /etc/hosts 2024-04-18T05:33:55.142 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.15.186 smithi186.front.sepia.ceph.com smithi186 2024-04-18T05:33:55.143 DEBUG:teuthology.orchestra.run.smithi186:> sudo hostname smithi186 2024-04-18T05:33:55.253 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i -e 's/smithi186.front.sepia.ceph.com/smithi186/g' /etc/hosts 2024-04-18T05:33:55.280 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i -e 's/172.21.15.186/172.21.15.186/g' /etc/hosts 2024-04-18T05:33:55.347 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i -e 's/smithi186.front.sepia.ceph.com/smithi186/g' /etc/hostname 2024-04-18T05:33:55.414 DEBUG:teuthology.orchestra.run.smithi186:> sudo hostname smithi186 2024-04-18T05:33:55.480 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2024-04-18T05:33:55.538 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="Red Hat Enterprise Linux" 2024-04-18T05:33:55.538 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="8.6 (Ootpa)" 2024-04-18T05:33:55.538 INFO:teuthology.orchestra.run.smithi186.stdout:ID="rhel" 2024-04-18T05:33:55.538 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="fedora" 2024-04-18T05:33:55.538 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="8.6" 2024-04-18T05:33:55.538 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el8" 2024-04-18T05:33:55.539 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-18T05:33:55.539 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2024-04-18T05:33:55.539 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-18T05:33:55.539 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://www.redhat.com/" 2024-04-18T05:33:55.539 INFO:teuthology.orchestra.run.smithi186.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-18T05:33:55.539 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-18T05:33:55.539 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-18T05:33:55.539 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-18T05:33:55.539 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-18T05:33:55.539 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-18T05:33:55.539 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-18T05:33:55.540 INFO:teuthology.provision.fog.smithi186:Deploy complete! 2024-04-18T05:33:55.541 DEBUG:teuthology.parallel:result is None 2024-04-18T05:34:03.038 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-04-18T05:34:09.039 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:34:12.098 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-04-18T05:34:18.100 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T05:34:49.839 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-04-18T05:34:49.840 DEBUG:teuthology.orchestra.run.smithi144:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-04-18T05:35:15.581 INFO:teuthology.provision.fog.smithi144:Node is ready 2024-04-18T05:35:15.581 DEBUG:teuthology.orchestra.run.smithi144:> hostname 2024-04-18T05:35:15.599 INFO:teuthology.orchestra.run.smithi144.stdout:smithi144 2024-04-18T05:35:15.600 DEBUG:teuthology.orchestra.run.smithi144:> grep smithi144 /etc/hosts 2024-04-18T05:35:15.657 INFO:teuthology.orchestra.run.smithi144.stdout:172.21.15.144 smithi144 smithi144 2024-04-18T05:35:15.658 DEBUG:teuthology.orchestra.run.smithi144:> sudo hostname smithi144 2024-04-18T05:35:15.725 DEBUG:teuthology.orchestra.run.smithi144:> sudo sed -i -e 's/smithi144/smithi144/g' /etc/hosts 2024-04-18T05:35:15.816 DEBUG:teuthology.orchestra.run.smithi144:> sudo sed -i -e 's/172.21.15.144/172.21.15.144/g' /etc/hosts 2024-04-18T05:35:15.843 DEBUG:teuthology.orchestra.run.smithi144:> sudo sed -i -e 's/smithi144/smithi144/g' /etc/hostname 2024-04-18T05:35:15.912 DEBUG:teuthology.orchestra.run.smithi144:> sudo hostname smithi144 2024-04-18T05:35:15.979 DEBUG:teuthology.orchestra.run.smithi144:> cat /etc/os-release 2024-04-18T05:35:16.037 INFO:teuthology.orchestra.run.smithi144.stdout:NAME="CentOS Stream" 2024-04-18T05:35:16.038 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION="8" 2024-04-18T05:35:16.038 INFO:teuthology.orchestra.run.smithi144.stdout:ID="centos" 2024-04-18T05:35:16.038 INFO:teuthology.orchestra.run.smithi144.stdout:ID_LIKE="rhel fedora" 2024-04-18T05:35:16.038 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION_ID="8" 2024-04-18T05:35:16.038 INFO:teuthology.orchestra.run.smithi144.stdout:PLATFORM_ID="platform:el8" 2024-04-18T05:35:16.038 INFO:teuthology.orchestra.run.smithi144.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-18T05:35:16.038 INFO:teuthology.orchestra.run.smithi144.stdout:ANSI_COLOR="0;31" 2024-04-18T05:35:16.038 INFO:teuthology.orchestra.run.smithi144.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-18T05:35:16.038 INFO:teuthology.orchestra.run.smithi144.stdout:HOME_URL="https://centos.org/" 2024-04-18T05:35:16.038 INFO:teuthology.orchestra.run.smithi144.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-18T05:35:16.038 INFO:teuthology.orchestra.run.smithi144.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-18T05:35:16.039 INFO:teuthology.orchestra.run.smithi144.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-18T05:35:16.041 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/provision/fog.py", line 92, in create self._verify_installed_os() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/provision/fog.py", line 347, in _verify_installed_os raise RuntimeError( RuntimeError: Expected smithi144's OS to be rhel 8.6 but found centos 8 2024-04-18T05:35:16.042 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-18T05:35:16.355 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=661dc6feef344c3691276975bc8a0a7b Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/provision/fog.py", line 92, in create self._verify_installed_os() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/provision/fog.py", line 347, in _verify_installed_os raise RuntimeError( RuntimeError: Expected smithi144's OS to be rhel 8.6 but found centos 8 2024-04-18T05:35:16.358 INFO:teuthology.nuke:Checking targets against current locks 2024-04-18T05:35:16.425 DEBUG:teuthology.nuke:shortname: smithi144 2024-04-18T05:35:16.426 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-18T05:35:16.452 DEBUG:teuthology.nuke:shortname: smithi186 2024-04-18T05:35:16.452 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-18T05:35:16.456 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi144.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-17_22:24:03-fs-reef-distro-default-smithi/7661598', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-18 05:30:31.376844', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEs/NADi7NBWmf+Rnq9mmCI4yulEZfNtfeVi2BRZpHU/7EGA9nfbVArAZiAlDJXAxvgzgwA/TOWNb+cadZU6u+0='} 2024-04-18T05:35:16.481 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-17_22:24:03-fs-reef-distro-default-smithi/7661595', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-18 05:28:39.852968', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBxsWT2wNY4V78LBHJ9Jzh4shoowQU9naZNVYxxNuFBQ6EnYK6obrqNQ0TwJKS+DqBc40yI343tq/Prs/TtgNzs='} 2024-04-18T05:35:16.513 INFO:teuthology.orchestra.console.smithi144:Power off 2024-04-18T05:35:16.514 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-18T05:35:16.537 DEBUG:teuthology.orchestra.console.smithi144:power off output: Chassis Power Control: Down/Off 2024-04-18T05:35:16.537 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T05:35:16.552 INFO:teuthology.orchestra.console.smithi186:Power off 2024-04-18T05:35:16.553 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-18T05:35:16.563 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is on 2024-04-18T05:35:16.581 DEBUG:teuthology.orchestra.console.smithi186:power off output: Chassis Power Control: Down/Off 2024-04-18T05:35:16.581 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T05:35:16.614 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-04-18T05:35:20.565 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T05:35:20.616 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T05:35:20.675 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is on 2024-04-18T05:35:20.726 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-04-18T05:35:24.676 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T05:35:24.726 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T05:35:24.787 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is off 2024-04-18T05:35:24.838 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-04-18T05:35:24.888 INFO:teuthology.orchestra.console.smithi144:Power off completed 2024-04-18T05:35:25.041 INFO:teuthology.lock.ops:unlocked: smithi144.front.sepia.ceph.com 2024-04-18T05:35:28.839 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T05:35:28.958 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is off 2024-04-18T05:35:29.059 INFO:teuthology.orchestra.console.smithi186:Power off completed 2024-04-18T05:35:29.215 INFO:teuthology.lock.ops:unlocked: smithi186.front.sepia.ceph.com 2024-04-18T05:35:29.489 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/dispatcher/supervisor.py", line 58, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/provision/fog.py", line 92, in create self._verify_installed_os() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/provision/fog.py", line 347, in _verify_installed_os raise RuntimeError( RuntimeError: Expected smithi144's OS to be rhel 8.6 but found centos 8