2024-04-06T04:33:42.471 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-06T04:33:42.471 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T04:33:42.616 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f51a4bc4c10>, signals=[15]) 2024-04-06T04:33:42.616 INFO:teuthology.lock.ops:Start node 'smithi105.front.sepia.ceph.com' reimaging 2024-04-06T04:33:42.617 INFO:teuthology.lock.ops:Updating [smithi105.front.sepia.ceph.com]: reset os type and version on server 2024-04-06T04:33:42.617 INFO:teuthology.lock.ops:Updating smithi105.front.sepia.ceph.com on lock server 2024-04-06T04:33:42.637 INFO:teuthology.lock.ops:Start node 'smithi181.front.sepia.ceph.com' reimaging 2024-04-06T04:33:42.637 INFO:teuthology.lock.ops:Updating [smithi181.front.sepia.ceph.com]: reset os type and version on server 2024-04-06T04:33:42.638 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2024-04-06T04:33:42.761 DEBUG:teuthology.provision.fog.smithi105:Requesting image smithi_centos_9.stream (ID 10) 2024-04-06T04:33:42.780 DEBUG:teuthology.provision.fog.smithi181:Requesting image smithi_centos_9.stream (ID 10) 2024-04-06T04:33:42.810 INFO:teuthology.provision.fog.smithi105:Scheduling deploy of centos 9.stream 2024-04-06T04:33:42.820 INFO:teuthology.provision.fog.smithi181:Scheduling deploy of centos 9.stream 2024-04-06T04:33:43.084 INFO:teuthology.orchestra.console.smithi105:Power off 2024-04-06T04:33:43.084 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-06T04:33:43.097 INFO:teuthology.orchestra.console.smithi181:Power off 2024-04-06T04:33:43.097 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-06T04:33:43.108 DEBUG:teuthology.orchestra.console.smithi105:power off output: Chassis Power Control: Down/Off 2024-04-06T04:33:43.108 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T04:33:43.119 DEBUG:teuthology.orchestra.console.smithi181:power off output: Chassis Power Control: Down/Off 2024-04-06T04:33:43.119 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T04:33:43.130 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on 2024-04-06T04:33:43.144 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is on 2024-04-06T04:33:47.134 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T04:33:47.145 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T04:33:47.245 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on 2024-04-06T04:33:47.254 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is on 2024-04-06T04:33:51.246 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T04:33:51.254 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T04:33:51.356 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on 2024-04-06T04:33:51.362 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is on 2024-04-06T04:33:55.358 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T04:33:55.365 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T04:33:55.467 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is off 2024-04-06T04:33:55.473 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is off 2024-04-06T04:33:55.569 INFO:teuthology.orchestra.console.smithi105:Power off completed 2024-04-06T04:33:55.574 INFO:teuthology.orchestra.console.smithi181:Power off completed 2024-04-06T04:33:55.670 INFO:teuthology.orchestra.console.smithi105:Power on 2024-04-06T04:33:55.670 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-04-06T04:33:55.677 INFO:teuthology.orchestra.console.smithi181:Power on 2024-04-06T04:33:55.677 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-04-06T04:33:55.691 DEBUG:teuthology.orchestra.console.smithi105:power on output: Chassis Power Control: Up/On 2024-04-06T04:33:55.691 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T04:33:55.699 DEBUG:teuthology.orchestra.console.smithi181:power on output: Chassis Power Control: Up/On 2024-04-06T04:33:55.699 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T04:33:55.711 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is off 2024-04-06T04:33:55.720 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is off 2024-04-06T04:33:59.717 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T04:33:59.725 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T04:33:59.827 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on 2024-04-06T04:33:59.834 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is on 2024-04-06T04:33:59.928 INFO:teuthology.orchestra.console.smithi105:Power on completed 2024-04-06T04:33:59.934 INFO:teuthology.orchestra.console.smithi181:Power on completed 2024-04-06T04:34:00.029 INFO:teuthology.provision.fog.smithi105:Waiting for deploy to finish 2024-04-06T04:34:00.035 INFO:teuthology.provision.fog.smithi181:Waiting for deploy to finish 2024-04-06T04:36:31.051 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T04:36:31.055 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T04:36:31.194 ERROR:teuthology.orchestra.connection:Error authenticating with smithi105.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2024-04-06T04:37:35.197 WARNING:teuthology.provision.fog.smithi105:timed out 2024-04-06T04:37:41.199 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T04:37:59.710 WARNING:teuthology.provision.fog.smithi105:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-04-06T04:38:01.223 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: AuthenticationException('Authentication timeout.') 2024-04-06T04:38:05.711 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T04:38:08.766 WARNING:teuthology.provision.fog.smithi105:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-04-06T04:38:14.768 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T04:38:15.261 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-04-06T04:38:15.262 DEBUG:teuthology.orchestra.run.smithi105:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-04-06T04:38:20.740 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: BadHostKeyException('smithi181.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:o8s+1gjJZep2iiS+K6WmRWw599KYveJKXslCD9l5hMA), PKey(alg=ED25519, bits=256, fp=SHA256:yY2VhCew8lbby+euI+Wuuvpz8gvXh54UvybB5xO1de8)) (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 468, in connect raise BadHostKeyException(hostname, server_key, our_key) paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi181.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAjFABXHj3+Cgqb3tTgWV9CLpqd4lSSOCmoYjqKh+exJM6pVTnTqf2B08CyDUqd7mqt/sxNPt1tzCuZVBn8XfQ0=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAIO8CdWiZW9gvDdM6E+3e3MFaz2XA5gssy4kb40cjUw7K' 2024-04-06T04:38:22.238 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-04-06T04:38:27.760 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: BadHostKeyException('smithi181.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:o8s+1gjJZep2iiS+K6WmRWw599KYveJKXslCD9l5hMA), PKey(alg=ED25519, bits=256, fp=SHA256:yY2VhCew8lbby+euI+Wuuvpz8gvXh54UvybB5xO1de8)) (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 468, in connect raise BadHostKeyException(hostname, server_key, our_key) paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi181.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAjFABXHj3+Cgqb3tTgWV9CLpqd4lSSOCmoYjqKh+exJM6pVTnTqf2B08CyDUqd7mqt/sxNPt1tzCuZVBn8XfQ0=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAIO8CdWiZW9gvDdM6E+3e3MFaz2XA5gssy4kb40cjUw7K' 2024-04-06T04:38:35.590 INFO:teuthology.provision.fog.smithi105:Node is ready 2024-04-06T04:38:35.591 DEBUG:teuthology.orchestra.run.smithi105:> hostname 2024-04-06T04:38:35.609 INFO:teuthology.orchestra.run.smithi105.stdout:smithi046 2024-04-06T04:38:35.609 DEBUG:teuthology.orchestra.run.smithi105:> grep smithi046 /etc/hosts 2024-04-06T04:38:35.666 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.46 smithi046.front.sepia.ceph.com smithi046 2024-04-06T04:38:35.667 DEBUG:teuthology.orchestra.run.smithi105:> sudo hostname smithi105 2024-04-06T04:38:35.859 DEBUG:teuthology.orchestra.run.smithi105:> sudo sed -i -e 's/smithi046/smithi105/g' /etc/hosts 2024-04-06T04:38:35.889 DEBUG:teuthology.orchestra.run.smithi105:> sudo sed -i -e 's/172.21.15.46/172.21.15.105/g' /etc/hosts 2024-04-06T04:38:35.959 DEBUG:teuthology.orchestra.run.smithi105:> sudo sed -i -e 's/smithi046/smithi105/g' /etc/hostname 2024-04-06T04:38:36.032 DEBUG:teuthology.orchestra.run.smithi105:> sudo hostname smithi105 2024-04-06T04:38:36.100 DEBUG:teuthology.orchestra.run.smithi105:> cat /etc/os-release 2024-04-06T04:38:36.154 INFO:teuthology.orchestra.run.smithi105.stdout:NAME="CentOS Stream" 2024-04-06T04:38:36.154 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION="9" 2024-04-06T04:38:36.154 INFO:teuthology.orchestra.run.smithi105.stdout:ID="centos" 2024-04-06T04:38:36.154 INFO:teuthology.orchestra.run.smithi105.stdout:ID_LIKE="rhel fedora" 2024-04-06T04:38:36.155 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION_ID="9" 2024-04-06T04:38:36.155 INFO:teuthology.orchestra.run.smithi105.stdout:PLATFORM_ID="platform:el9" 2024-04-06T04:38:36.155 INFO:teuthology.orchestra.run.smithi105.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-06T04:38:36.155 INFO:teuthology.orchestra.run.smithi105.stdout:ANSI_COLOR="0;31" 2024-04-06T04:38:36.155 INFO:teuthology.orchestra.run.smithi105.stdout:LOGO="fedora-logo-icon" 2024-04-06T04:38:36.155 INFO:teuthology.orchestra.run.smithi105.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-06T04:38:36.155 INFO:teuthology.orchestra.run.smithi105.stdout:HOME_URL="https://centos.org/" 2024-04-06T04:38:36.155 INFO:teuthology.orchestra.run.smithi105.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-06T04:38:36.155 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-06T04:38:36.155 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-06T04:38:36.156 INFO:teuthology.provision.fog.smithi105:Deploy complete! 2024-04-06T04:38:36.157 DEBUG:teuthology.parallel:result is None 2024-04-06T04:38:37.825 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: BadHostKeyException('smithi181.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:o8s+1gjJZep2iiS+K6WmRWw599KYveJKXslCD9l5hMA), PKey(alg=ED25519, bits=256, fp=SHA256:yY2VhCew8lbby+euI+Wuuvpz8gvXh54UvybB5xO1de8)) (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 468, in connect raise BadHostKeyException(hostname, server_key, our_key) paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi181.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAjFABXHj3+Cgqb3tTgWV9CLpqd4lSSOCmoYjqKh+exJM6pVTnTqf2B08CyDUqd7mqt/sxNPt1tzCuZVBn8XfQ0=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAIO8CdWiZW9gvDdM6E+3e3MFaz2XA5gssy4kb40cjUw7K' 2024-04-06T04:38:50.843 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: BadHostKeyException('smithi181.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:o8s+1gjJZep2iiS+K6WmRWw599KYveJKXslCD9l5hMA), PKey(alg=ED25519, bits=256, fp=SHA256:yY2VhCew8lbby+euI+Wuuvpz8gvXh54UvybB5xO1de8)) (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 468, in connect raise BadHostKeyException(hostname, server_key, our_key) paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi181.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAjFABXHj3+Cgqb3tTgWV9CLpqd4lSSOCmoYjqKh+exJM6pVTnTqf2B08CyDUqd7mqt/sxNPt1tzCuZVBn8XfQ0=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAIO8CdWiZW9gvDdM6E+3e3MFaz2XA5gssy4kb40cjUw7K' 2024-04-06T04:39:06.863 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: BadHostKeyException('smithi181.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:o8s+1gjJZep2iiS+K6WmRWw599KYveJKXslCD9l5hMA), PKey(alg=ED25519, bits=256, fp=SHA256:yY2VhCew8lbby+euI+Wuuvpz8gvXh54UvybB5xO1de8)) (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 468, in connect raise BadHostKeyException(hostname, server_key, our_key) paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi181.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAjFABXHj3+Cgqb3tTgWV9CLpqd4lSSOCmoYjqKh+exJM6pVTnTqf2B08CyDUqd7mqt/sxNPt1tzCuZVBn8XfQ0=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAIO8CdWiZW9gvDdM6E+3e3MFaz2XA5gssy4kb40cjUw7K' 2024-04-06T04:39:25.882 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: BadHostKeyException('smithi181.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:o8s+1gjJZep2iiS+K6WmRWw599KYveJKXslCD9l5hMA), PKey(alg=ED25519, bits=256, fp=SHA256:yY2VhCew8lbby+euI+Wuuvpz8gvXh54UvybB5xO1de8)) (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 468, in connect raise BadHostKeyException(hostname, server_key, our_key) paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi181.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAjFABXHj3+Cgqb3tTgWV9CLpqd4lSSOCmoYjqKh+exJM6pVTnTqf2B08CyDUqd7mqt/sxNPt1tzCuZVBn8XfQ0=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAIO8CdWiZW9gvDdM6E+3e3MFaz2XA5gssy4kb40cjUw7K' 2024-04-06T04:39:47.901 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: BadHostKeyException('smithi181.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:o8s+1gjJZep2iiS+K6WmRWw599KYveJKXslCD9l5hMA), PKey(alg=ED25519, bits=256, fp=SHA256:yY2VhCew8lbby+euI+Wuuvpz8gvXh54UvybB5xO1de8)) (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 468, in connect raise BadHostKeyException(hostname, server_key, our_key) paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi181.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAjFABXHj3+Cgqb3tTgWV9CLpqd4lSSOCmoYjqKh+exJM6pVTnTqf2B08CyDUqd7mqt/sxNPt1tzCuZVBn8XfQ0=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAIO8CdWiZW9gvDdM6E+3e3MFaz2XA5gssy4kb40cjUw7K' 2024-04-06T04:40:12.921 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: BadHostKeyException('smithi181.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:o8s+1gjJZep2iiS+K6WmRWw599KYveJKXslCD9l5hMA), PKey(alg=ED25519, bits=256, fp=SHA256:yY2VhCew8lbby+euI+Wuuvpz8gvXh54UvybB5xO1de8)) (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 468, in connect raise BadHostKeyException(hostname, server_key, our_key) paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi181.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAjFABXHj3+Cgqb3tTgWV9CLpqd4lSSOCmoYjqKh+exJM6pVTnTqf2B08CyDUqd7mqt/sxNPt1tzCuZVBn8XfQ0=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAIO8CdWiZW9gvDdM6E+3e3MFaz2XA5gssy4kb40cjUw7K' 2024-04-06T04:40:40.941 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: BadHostKeyException('smithi181.front.sepia.ceph.com', PKey(alg=ECDSA, bits=256, fp=SHA256:o8s+1gjJZep2iiS+K6WmRWw599KYveJKXslCD9l5hMA), PKey(alg=ED25519, bits=256, fp=SHA256:yY2VhCew8lbby+euI+Wuuvpz8gvXh54UvybB5xO1de8)) (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 468, in connect raise BadHostKeyException(hostname, server_key, our_key) paramiko.ssh_exception.BadHostKeyException: Host key for server 'smithi181.front.sepia.ceph.com' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAjFABXHj3+Cgqb3tTgWV9CLpqd4lSSOCmoYjqKh+exJM6pVTnTqf2B08CyDUqd7mqt/sxNPt1tzCuZVBn8XfQ0=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAIO8CdWiZW9gvDdM6E+3e3MFaz2XA5gssy4kb40cjUw7K' 2024-04-06T04:40:40.942 WARNING:teuthology.provision.fog.smithi181:'connect to smithi181.front.sepia.ceph.com' reached maximum tries (11) after waiting for 145 seconds 2024-04-06T04:40:46.943 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T04:40:47.161 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-04-06T04:40:47.161 DEBUG:teuthology.orchestra.run.smithi181:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-04-06T04:40:47.475 INFO:teuthology.provision.fog.smithi181:Node is ready 2024-04-06T04:40:47.476 DEBUG:teuthology.orchestra.run.smithi181:> hostname 2024-04-06T04:40:47.490 INFO:teuthology.orchestra.run.smithi181.stdout:smithi181.front.sepia.ceph.com 2024-04-06T04:40:47.491 DEBUG:teuthology.orchestra.run.smithi181:> grep smithi181.front.sepia.ceph.com /etc/hosts 2024-04-06T04:40:47.546 INFO:teuthology.orchestra.run.smithi181.stdout:172.21.15.181 smithi181.front.sepia.ceph.com smithi181 2024-04-06T04:40:47.546 DEBUG:teuthology.orchestra.run.smithi181:> sudo hostname smithi181 2024-04-06T04:40:47.660 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i -e 's/smithi181.front.sepia.ceph.com/smithi181/g' /etc/hosts 2024-04-06T04:40:47.687 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i -e 's/172.21.15.181/172.21.15.181/g' /etc/hosts 2024-04-06T04:40:47.757 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i -e 's/smithi181.front.sepia.ceph.com/smithi181/g' /etc/hostname 2024-04-06T04:40:47.822 DEBUG:teuthology.orchestra.run.smithi181:> sudo hostname smithi181 2024-04-06T04:40:47.888 DEBUG:teuthology.orchestra.run.smithi181:> cat /etc/os-release 2024-04-06T04:40:47.944 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="CentOS Stream" 2024-04-06T04:40:47.944 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="9" 2024-04-06T04:40:47.944 INFO:teuthology.orchestra.run.smithi181.stdout:ID="centos" 2024-04-06T04:40:47.944 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="rhel fedora" 2024-04-06T04:40:47.944 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="9" 2024-04-06T04:40:47.944 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el9" 2024-04-06T04:40:47.944 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-06T04:40:47.944 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31" 2024-04-06T04:40:47.944 INFO:teuthology.orchestra.run.smithi181.stdout:LOGO="fedora-logo-icon" 2024-04-06T04:40:47.945 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-06T04:40:47.945 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://centos.org/" 2024-04-06T04:40:47.945 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-06T04:40:47.945 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-06T04:40:47.945 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-06T04:40:47.945 INFO:teuthology.provision.fog.smithi181:Deploy complete! 2024-04-06T04:40:47.946 DEBUG:teuthology.parallel:result is None 2024-04-06T04:40:48.121 INFO:teuthology.lock.ops:Checking smithi105.front.sepia.ceph.com 2024-04-06T04:40:48.121 INFO:teuthology.lock.ops:New key found. Updating... 2024-04-06T04:40:48.146 INFO:teuthology.lock.ops:Checking smithi181.front.sepia.ceph.com 2024-04-06T04:40:48.146 INFO:teuthology.lock.ops:New key found. Updating... 2024-04-06T04:40:48.166 INFO:teuthology.lock.ops:Updating [smithi105.front.sepia.ceph.com]: set os type and version on server 2024-04-06T04:40:48.166 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-04-06T04:40:48.166 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T04:40:48.258 DEBUG:teuthology.orchestra.run.smithi105:> true 2024-04-06T04:40:48.327 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-04-06T04:40:48.328 DEBUG:teuthology.orchestra.run.smithi105:> uname -m 2024-04-06T04:40:48.383 INFO:teuthology.orchestra.run.smithi105.stdout:x86_64 2024-04-06T04:40:48.383 DEBUG:teuthology.orchestra.run.smithi105:> cat /etc/os-release 2024-04-06T04:40:48.439 INFO:teuthology.orchestra.run.smithi105.stdout:NAME="CentOS Stream" 2024-04-06T04:40:48.439 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION="9" 2024-04-06T04:40:48.439 INFO:teuthology.orchestra.run.smithi105.stdout:ID="centos" 2024-04-06T04:40:48.439 INFO:teuthology.orchestra.run.smithi105.stdout:ID_LIKE="rhel fedora" 2024-04-06T04:40:48.439 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION_ID="9" 2024-04-06T04:40:48.439 INFO:teuthology.orchestra.run.smithi105.stdout:PLATFORM_ID="platform:el9" 2024-04-06T04:40:48.440 INFO:teuthology.orchestra.run.smithi105.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-06T04:40:48.440 INFO:teuthology.orchestra.run.smithi105.stdout:ANSI_COLOR="0;31" 2024-04-06T04:40:48.440 INFO:teuthology.orchestra.run.smithi105.stdout:LOGO="fedora-logo-icon" 2024-04-06T04:40:48.440 INFO:teuthology.orchestra.run.smithi105.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-06T04:40:48.440 INFO:teuthology.orchestra.run.smithi105.stdout:HOME_URL="https://centos.org/" 2024-04-06T04:40:48.440 INFO:teuthology.orchestra.run.smithi105.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-06T04:40:48.440 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-06T04:40:48.440 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-06T04:40:48.440 INFO:teuthology.lock.ops:Updating smithi105.front.sepia.ceph.com on lock server 2024-04-06T04:40:48.464 INFO:teuthology.lock.ops:Updating [smithi181.front.sepia.ceph.com]: set os type and version on server 2024-04-06T04:40:48.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-04-06T04:40:48.465 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T04:40:48.552 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-04-06T04:40:48.620 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-04-06T04:40:48.620 DEBUG:teuthology.orchestra.run.smithi181:> uname -m 2024-04-06T04:40:48.674 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64 2024-04-06T04:40:48.675 DEBUG:teuthology.orchestra.run.smithi181:> cat /etc/os-release 2024-04-06T04:40:48.729 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="CentOS Stream" 2024-04-06T04:40:48.729 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="9" 2024-04-06T04:40:48.730 INFO:teuthology.orchestra.run.smithi181.stdout:ID="centos" 2024-04-06T04:40:48.730 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="rhel fedora" 2024-04-06T04:40:48.730 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="9" 2024-04-06T04:40:48.730 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el9" 2024-04-06T04:40:48.730 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-06T04:40:48.730 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31" 2024-04-06T04:40:48.730 INFO:teuthology.orchestra.run.smithi181.stdout:LOGO="fedora-logo-icon" 2024-04-06T04:40:48.730 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-06T04:40:48.730 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://centos.org/" 2024-04-06T04:40:48.730 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-06T04:40:48.730 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-06T04:40:48.730 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-06T04:40:48.731 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2024-04-06T04:40:48.755 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T04:40:48.849 INFO:teuthology.dispatcher.supervisor:Running job 7642352 2024-04-06T04:40:48.850 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-04-06_03:08:13-upgrade-main-distro-default-smithi/7642352 --name teuthology-2024-04-06_03:08:13-upgrade-main-distro-default-smithi --description upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} -- /home/teuthworker/archive/teuthology-2024-04-06_03:08:13-upgrade-main-distro-default-smithi/7642352/orig.config.yaml 2024-04-06T04:40:48.853 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2024-04-06_03:08:13-upgrade-main-distro-default-smithi/7642352 2024-04-06T04:40:48.853 INFO:teuthology.dispatcher.supervisor:Job PID: 458108 2024-04-06T04:40:48.854 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2024-04-06T04:42:48.855 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T04:44:48.941 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T04:46:49.018 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T04:48:49.094 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T04:50:49.159 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T04:52:49.225 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T04:54:49.301 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T04:56:49.370 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T04:58:49.440 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T05:00:49.514 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T05:02:49.588 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T05:04:49.651 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T05:06:49.727 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T05:08:49.802 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T05:10:49.882 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T05:12:49.948 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T05:14:50.023 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T05:16:50.103 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T05:18:50.175 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T05:20:50.248 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T05:22:50.324 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T05:24:50.397 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T05:26:50.472 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T05:28:50.546 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T05:30:50.620 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T05:32:50.697 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T05:34:50.764 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T05:36:50.840 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T05:38:50.912 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T05:40:50.978 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T05:42:51.046 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T05:44:51.118 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T05:46:51.194 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T05:48:51.260 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T05:48:51.326 ERROR:teuthology.dispatcher.supervisor:Child exited with code 1 2024-04-06T05:48:51.443 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi105 but it was locked by another job: /home/teuthworker/archive/teuthology-2024-04-01_22:08:02-orch-reef-distro-default-smithi/7635121