2024-03-28T12:34:58.672 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-28T12:34:58.673 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T12:34:58.834 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbed09a3c10>, signals=[15]) 2024-03-28T12:34:58.834 INFO:teuthology.lock.ops:Start node 'smithi050.front.sepia.ceph.com' reimaging 2024-03-28T12:34:58.834 INFO:teuthology.lock.ops:Updating [smithi050.front.sepia.ceph.com]: reset os type and version on server 2024-03-28T12:34:58.835 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server 2024-03-28T12:34:58.852 INFO:teuthology.lock.ops:Start node 'smithi163.front.sepia.ceph.com' reimaging 2024-03-28T12:34:58.852 INFO:teuthology.lock.ops:Updating [smithi163.front.sepia.ceph.com]: reset os type and version on server 2024-03-28T12:34:58.853 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server 2024-03-28T12:34:58.868 INFO:teuthology.lock.ops:Start node 'smithi169.front.sepia.ceph.com' reimaging 2024-03-28T12:34:58.869 INFO:teuthology.lock.ops:Updating [smithi169.front.sepia.ceph.com]: reset os type and version on server 2024-03-28T12:34:58.869 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server 2024-03-28T12:34:58.979 DEBUG:teuthology.provision.fog.smithi050:Requesting image smithi_centos_8.stream (ID 3) 2024-03-28T12:34:58.982 DEBUG:teuthology.provision.fog.smithi163:Requesting image smithi_centos_8.stream (ID 3) 2024-03-28T12:34:58.996 DEBUG:teuthology.provision.fog.smithi169:Requesting image smithi_centos_8.stream (ID 3) 2024-03-28T12:34:59.020 INFO:teuthology.provision.fog.smithi050:Scheduling deploy of centos 8.stream 2024-03-28T12:34:59.030 INFO:teuthology.provision.fog.smithi163:Scheduling deploy of centos 8.stream 2024-03-28T12:34:59.043 INFO:teuthology.provision.fog.smithi169:Scheduling deploy of centos 8.stream 2024-03-28T12:34:59.111 DEBUG:teuthology.provision.fog.smithi169:Canceling deploy task with ID 902191 2024-03-28T12:34:59.330 INFO:teuthology.orchestra.console.smithi163:Power off 2024-03-28T12:34:59.330 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T12:34:59.343 INFO:teuthology.orchestra.console.smithi050:Power off 2024-03-28T12:34:59.344 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T12:34:59.352 DEBUG:teuthology.orchestra.console.smithi163:power off output: Chassis Power Control: Down/Off 2024-03-28T12:34:59.353 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T12:34:59.365 DEBUG:teuthology.orchestra.console.smithi050:power off output: Chassis Power Control: Down/Off 2024-03-28T12:34:59.366 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T12:34:59.373 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is off 2024-03-28T12:34:59.385 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is off 2024-03-28T12:34:59.473 INFO:teuthology.orchestra.console.smithi163:Power off completed 2024-03-28T12:34:59.491 INFO:teuthology.orchestra.console.smithi169:Power off 2024-03-28T12:34:59.491 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T12:34:59.497 INFO:teuthology.orchestra.console.smithi050:Power off completed 2024-03-28T12:34:59.512 DEBUG:teuthology.orchestra.console.smithi169:power off output: Chassis Power Control: Down/Off 2024-03-28T12:34:59.513 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T12:34:59.540 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is off 2024-03-28T12:34:59.574 INFO:teuthology.orchestra.console.smithi163:Power on 2024-03-28T12:34:59.574 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-28T12:34:59.598 DEBUG:teuthology.orchestra.console.smithi163:power on output: Chassis Power Control: Up/On 2024-03-28T12:34:59.599 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T12:34:59.607 INFO:teuthology.orchestra.console.smithi050:Power on 2024-03-28T12:34:59.607 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-28T12:34:59.620 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is off 2024-03-28T12:34:59.629 DEBUG:teuthology.orchestra.console.smithi050:power on output: Chassis Power Control: Up/On 2024-03-28T12:34:59.630 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T12:34:59.642 INFO:teuthology.orchestra.console.smithi169:Power off completed 2024-03-28T12:34:59.653 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is off 2024-03-28T12:34:59.743 INFO:teuthology.orchestra.console.smithi169:Power on 2024-03-28T12:34:59.744 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-28T12:34:59.770 DEBUG:teuthology.orchestra.console.smithi169:power on output: Chassis Power Control: Up/On 2024-03-28T12:34:59.776 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T12:34:59.798 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is off 2024-03-28T12:35:03.622 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T12:35:03.654 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T12:35:03.731 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2024-03-28T12:35:03.763 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is on 2024-03-28T12:35:03.799 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T12:35:03.832 INFO:teuthology.orchestra.console.smithi163:Power on completed 2024-03-28T12:35:03.864 INFO:teuthology.orchestra.console.smithi050:Power on completed 2024-03-28T12:35:03.908 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is off 2024-03-28T12:35:03.933 INFO:teuthology.provision.fog.smithi163:Waiting for deploy to finish 2024-03-28T12:35:03.965 INFO:teuthology.provision.fog.smithi050:Waiting for deploy to finish 2024-03-28T12:35:07.910 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T12:35:08.020 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-03-28T12:35:08.121 INFO:teuthology.orchestra.console.smithi169:Power on completed 2024-03-28T12:35:08.221 INFO:teuthology.provision.fog.smithi169:Waiting for deploy to finish 2024-03-28T12:37:51.759 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:37:54.814 WARNING:teuthology.provision.fog.smithi050:[Errno None] Unable to connect to port 22 on 172.21.15.50 2024-03-28T12:38:00.820 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:38:03.870 WARNING:teuthology.provision.fog.smithi050:[Errno None] Unable to connect to port 22 on 172.21.15.50 2024-03-28T12:38:09.874 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:38:12.926 WARNING:teuthology.provision.fog.smithi050:[Errno None] Unable to connect to port 22 on 172.21.15.50 2024-03-28T12:38:18.932 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:38:22.014 WARNING:teuthology.provision.fog.smithi050:[Errno None] Unable to connect to port 22 on 172.21.15.50 2024-03-28T12:38:28.016 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:38:31.070 WARNING:teuthology.provision.fog.smithi050:[Errno None] Unable to connect to port 22 on 172.21.15.50 2024-03-28T12:38:37.078 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:38:42.847 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:39:17.534 WARNING:teuthology.provision.fog.smithi169:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-03-28T12:39:23.537 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:39:26.591 WARNING:teuthology.provision.fog.smithi169:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-03-28T12:39:32.592 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:39:35.650 WARNING:teuthology.provision.fog.smithi169:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-03-28T12:39:37.081 WARNING:teuthology.provision.fog.smithi050:timed out 2024-03-28T12:39:41.655 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:39:43.083 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:39:43.322 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-03-28T12:39:43.322 DEBUG:teuthology.orchestra.run.smithi050:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-03-28T12:39:43.582 INFO:teuthology.provision.fog.smithi050:Node is ready 2024-03-28T12:39:43.582 DEBUG:teuthology.orchestra.run.smithi050:> hostname 2024-03-28T12:39:43.603 INFO:teuthology.orchestra.run.smithi050.stdout:smithi050.front.sepia.ceph.com 2024-03-28T12:39:43.603 DEBUG:teuthology.orchestra.run.smithi050:> grep smithi050.front.sepia.ceph.com /etc/hosts 2024-03-28T12:39:43.663 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.15.50 smithi050.front.sepia.ceph.com smithi050 2024-03-28T12:39:43.663 DEBUG:teuthology.orchestra.run.smithi050:> sudo hostname smithi050 2024-03-28T12:39:43.770 DEBUG:teuthology.orchestra.run.smithi050:> sudo sed -i -e 's/smithi050.front.sepia.ceph.com/smithi050/g' /etc/hosts 2024-03-28T12:39:43.800 DEBUG:teuthology.orchestra.run.smithi050:> sudo sed -i -e 's/172.21.15.50/172.21.15.50/g' /etc/hosts 2024-03-28T12:39:43.870 DEBUG:teuthology.orchestra.run.smithi050:> sudo sed -i -e 's/smithi050.front.sepia.ceph.com/smithi050/g' /etc/hostname 2024-03-28T12:39:43.940 DEBUG:teuthology.orchestra.run.smithi050:> sudo hostname smithi050 2024-03-28T12:39:44.010 DEBUG:teuthology.orchestra.run.smithi050:> cat /etc/os-release 2024-03-28T12:39:44.069 INFO:teuthology.orchestra.run.smithi050.stdout:NAME="CentOS Stream" 2024-03-28T12:39:44.070 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION="8" 2024-03-28T12:39:44.070 INFO:teuthology.orchestra.run.smithi050.stdout:ID="centos" 2024-03-28T12:39:44.070 INFO:teuthology.orchestra.run.smithi050.stdout:ID_LIKE="rhel fedora" 2024-03-28T12:39:44.070 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_ID="8" 2024-03-28T12:39:44.070 INFO:teuthology.orchestra.run.smithi050.stdout:PLATFORM_ID="platform:el8" 2024-03-28T12:39:44.070 INFO:teuthology.orchestra.run.smithi050.stdout:PRETTY_NAME="CentOS Stream 8" 2024-03-28T12:39:44.070 INFO:teuthology.orchestra.run.smithi050.stdout:ANSI_COLOR="0;31" 2024-03-28T12:39:44.070 INFO:teuthology.orchestra.run.smithi050.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-03-28T12:39:44.070 INFO:teuthology.orchestra.run.smithi050.stdout:HOME_URL="https://centos.org/" 2024-03-28T12:39:44.070 INFO:teuthology.orchestra.run.smithi050.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T12:39:44.070 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-28T12:39:44.070 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T12:39:44.071 INFO:teuthology.provision.fog.smithi050:Deploy complete! 2024-03-28T12:39:44.072 DEBUG:teuthology.parallel:result is None 2024-03-28T12:40:00.030 WARNING:teuthology.provision.fog.smithi169:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-03-28T12:40:06.035 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:40:09.090 WARNING:teuthology.provision.fog.smithi169:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-03-28T12:40:15.097 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:40:16.363 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-03-28T12:40:16.364 DEBUG:teuthology.orchestra.run.smithi169:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-03-28T12:40:41.675 INFO:teuthology.provision.fog.smithi169:Node is ready 2024-03-28T12:40:41.676 DEBUG:teuthology.orchestra.run.smithi169:> hostname 2024-03-28T12:40:41.736 INFO:teuthology.orchestra.run.smithi169.stdout:smithi169.front.sepia.ceph.com 2024-03-28T12:40:41.736 DEBUG:teuthology.orchestra.run.smithi169:> grep smithi169.front.sepia.ceph.com /etc/hosts 2024-03-28T12:40:41.795 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169 smithi169.front.sepia.ceph.com smithi169 2024-03-28T12:40:41.795 DEBUG:teuthology.orchestra.run.smithi169:> sudo hostname smithi169 2024-03-28T12:40:41.950 DEBUG:teuthology.orchestra.run.smithi169:> sudo sed -i -e 's/smithi169.front.sepia.ceph.com/smithi169/g' /etc/hosts 2024-03-28T12:40:41.997 DEBUG:teuthology.orchestra.run.smithi169:> sudo sed -i -e 's/172.21.15.169/172.21.15.169/g' /etc/hosts 2024-03-28T12:40:42.025 DEBUG:teuthology.orchestra.run.smithi169:> sudo sed -i -e 's/smithi169.front.sepia.ceph.com/smithi169/g' /etc/hostname 2024-03-28T12:40:42.093 DEBUG:teuthology.orchestra.run.smithi169:> sudo hostname smithi169 2024-03-28T12:40:42.160 DEBUG:teuthology.orchestra.run.smithi169:> cat /etc/os-release 2024-03-28T12:40:42.219 INFO:teuthology.orchestra.run.smithi169.stdout:NAME="CentOS Stream" 2024-03-28T12:40:42.220 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION="8" 2024-03-28T12:40:42.220 INFO:teuthology.orchestra.run.smithi169.stdout:ID="centos" 2024-03-28T12:40:42.220 INFO:teuthology.orchestra.run.smithi169.stdout:ID_LIKE="rhel fedora" 2024-03-28T12:40:42.220 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_ID="8" 2024-03-28T12:40:42.220 INFO:teuthology.orchestra.run.smithi169.stdout:PLATFORM_ID="platform:el8" 2024-03-28T12:40:42.220 INFO:teuthology.orchestra.run.smithi169.stdout:PRETTY_NAME="CentOS Stream 8" 2024-03-28T12:40:42.220 INFO:teuthology.orchestra.run.smithi169.stdout:ANSI_COLOR="0;31" 2024-03-28T12:40:42.220 INFO:teuthology.orchestra.run.smithi169.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-03-28T12:40:42.220 INFO:teuthology.orchestra.run.smithi169.stdout:HOME_URL="https://centos.org/" 2024-03-28T12:40:42.220 INFO:teuthology.orchestra.run.smithi169.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T12:40:42.220 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-28T12:40:42.220 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T12:40:42.221 INFO:teuthology.provision.fog.smithi169:Deploy complete! 2024-03-28T12:40:42.221 DEBUG:teuthology.parallel:result is None 2024-03-28T12:41:54.973 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:41:58.047 WARNING:teuthology.provision.fog.smithi163:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-28T12:42:04.049 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:42:07.106 WARNING:teuthology.provision.fog.smithi163:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-28T12:42:13.113 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:42:16.190 WARNING:teuthology.provision.fog.smithi163:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-28T12:42:22.194 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:42:25.246 WARNING:teuthology.provision.fog.smithi163:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-28T12:42:31.253 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:42:34.334 WARNING:teuthology.provision.fog.smithi163:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-28T12:42:40.341 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:42:43.422 WARNING:teuthology.provision.fog.smithi163:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-28T12:42:49.429 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:42:52.510 WARNING:teuthology.provision.fog.smithi163:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-28T12:42:58.515 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:43:01.566 WARNING:teuthology.provision.fog.smithi163:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-28T12:43:07.574 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:43:10.654 WARNING:teuthology.provision.fog.smithi163:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-28T12:43:16.659 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:44:16.720 WARNING:teuthology.provision.fog.smithi163:timed out 2024-03-28T12:44:22.728 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:44:22.988 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-03-28T12:44:22.988 DEBUG:teuthology.orchestra.run.smithi163:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-03-28T12:44:23.231 INFO:teuthology.provision.fog.smithi163:Node is ready 2024-03-28T12:44:23.231 DEBUG:teuthology.orchestra.run.smithi163:> hostname 2024-03-28T12:44:23.249 INFO:teuthology.orchestra.run.smithi163.stdout:smithi163.front.sepia.ceph.com 2024-03-28T12:44:23.250 DEBUG:teuthology.orchestra.run.smithi163:> grep smithi163.front.sepia.ceph.com /etc/hosts 2024-03-28T12:44:23.309 INFO:teuthology.orchestra.run.smithi163.stdout:172.21.15.163 smithi163.front.sepia.ceph.com smithi163 2024-03-28T12:44:23.309 DEBUG:teuthology.orchestra.run.smithi163:> sudo hostname smithi163 2024-03-28T12:44:23.485 DEBUG:teuthology.orchestra.run.smithi163:> sudo sed -i -e 's/smithi163.front.sepia.ceph.com/smithi163/g' /etc/hosts 2024-03-28T12:44:23.515 DEBUG:teuthology.orchestra.run.smithi163:> sudo sed -i -e 's/172.21.15.163/172.21.15.163/g' /etc/hosts 2024-03-28T12:44:23.584 DEBUG:teuthology.orchestra.run.smithi163:> sudo sed -i -e 's/smithi163.front.sepia.ceph.com/smithi163/g' /etc/hostname 2024-03-28T12:44:23.652 DEBUG:teuthology.orchestra.run.smithi163:> sudo hostname smithi163 2024-03-28T12:44:23.719 DEBUG:teuthology.orchestra.run.smithi163:> cat /etc/os-release 2024-03-28T12:44:23.777 INFO:teuthology.orchestra.run.smithi163.stdout:NAME="CentOS Stream" 2024-03-28T12:44:23.777 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION="8" 2024-03-28T12:44:23.777 INFO:teuthology.orchestra.run.smithi163.stdout:ID="centos" 2024-03-28T12:44:23.777 INFO:teuthology.orchestra.run.smithi163.stdout:ID_LIKE="rhel fedora" 2024-03-28T12:44:23.777 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION_ID="8" 2024-03-28T12:44:23.777 INFO:teuthology.orchestra.run.smithi163.stdout:PLATFORM_ID="platform:el8" 2024-03-28T12:44:23.778 INFO:teuthology.orchestra.run.smithi163.stdout:PRETTY_NAME="CentOS Stream 8" 2024-03-28T12:44:23.778 INFO:teuthology.orchestra.run.smithi163.stdout:ANSI_COLOR="0;31" 2024-03-28T12:44:23.778 INFO:teuthology.orchestra.run.smithi163.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-03-28T12:44:23.778 INFO:teuthology.orchestra.run.smithi163.stdout:HOME_URL="https://centos.org/" 2024-03-28T12:44:23.778 INFO:teuthology.orchestra.run.smithi163.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T12:44:23.778 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-28T12:44:23.778 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T12:44:23.778 INFO:teuthology.provision.fog.smithi163:Deploy complete! 2024-03-28T12:44:23.779 DEBUG:teuthology.parallel:result is None 2024-03-28T12:44:24.947 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 326, in reimage_machines reimaged = do_update_keys(list(reimaged.keys()))[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 290, in do_update_keys keys_dict = misc.ssh_keyscan(machines, _raise=_raise) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1108, in ssh_keyscan key = _ssh_keyscan(hostname) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1145, in _ssh_keyscan return sorted(keys)[0] IndexError: list index out of range 2024-03-28T12:44:24.949 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T12:44:25.252 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=5a224852ffa6456abcec52b4c9fe9397 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 326, in reimage_machines reimaged = do_update_keys(list(reimaged.keys()))[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 290, in do_update_keys keys_dict = misc.ssh_keyscan(machines, _raise=_raise) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1108, in ssh_keyscan key = _ssh_keyscan(hostname) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1145, in _ssh_keyscan return sorted(keys)[0] IndexError: list index out of range 2024-03-28T12:44:25.254 INFO:teuthology.nuke:Checking targets against current locks 2024-03-28T12:44:25.318 DEBUG:teuthology.nuke:shortname: smithi050 2024-03-28T12:44:25.318 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T12:44:25.349 DEBUG:teuthology.nuke:shortname: smithi163 2024-03-28T12:44:25.349 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T12:44:25.353 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627804', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 12:34:57.532789', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJlK/ogq+eYLm6lS+wefwO3Lq4Lh4DBGEXuON+ns/X+lRH1ORKErDXkEcMjcDu6tTLNYGBQ1y9Qi0XvIg6DYV5Y='} 2024-03-28T12:44:25.376 DEBUG:teuthology.nuke:shortname: smithi169 2024-03-28T12:44:25.376 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T12:44:25.381 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627804', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 12:34:57.534781', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCuRoMB5DSHJ5EeZ8LxICKdW0pfeLMdzON91YU9mQvXVpEfHsXzS29EOoDTLJgwKQvYhlphOaR0rJPxu+GyWG+8='} 2024-03-28T12:44:25.406 INFO:teuthology.orchestra.console.smithi050:Power off 2024-03-28T12:44:25.407 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T12:44:25.419 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-28_07:14:07-fs-wip-batrick-testing-20240327.230800-distro-default-smithi/7627848', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 12:43:56.452527', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDvqBRAdkinW6eathrE4TDlDVrjB27UcWoRVSmCzPzKFCw0tUqXvPEzss8vK65QBSeXuzWno0S74/iYffI/OgUM='} 2024-03-28T12:44:25.432 DEBUG:teuthology.orchestra.console.smithi050:power off output: Chassis Power Control: Down/Off 2024-03-28T12:44:25.432 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T12:44:25.453 INFO:teuthology.orchestra.console.smithi163:Power off 2024-03-28T12:44:25.454 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T12:44:25.467 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is on 2024-03-28T12:44:25.476 DEBUG:teuthology.orchestra.console.smithi163:power off output: Chassis Power Control: Down/Off 2024-03-28T12:44:25.476 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T12:44:25.489 INFO:teuthology.orchestra.console.smithi169:Power off 2024-03-28T12:44:25.490 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T12:44:25.499 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2024-03-28T12:44:25.511 DEBUG:teuthology.orchestra.console.smithi169:power off output: Chassis Power Control: Down/Off 2024-03-28T12:44:25.511 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T12:44:25.532 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-03-28T12:44:29.470 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T12:44:29.500 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T12:44:29.534 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T12:44:29.580 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is on 2024-03-28T12:44:29.609 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2024-03-28T12:44:29.644 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-03-28T12:44:33.585 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T12:44:33.610 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T12:44:33.645 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T12:44:33.694 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is off 2024-03-28T12:44:33.720 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2024-03-28T12:44:33.755 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-03-28T12:44:33.795 INFO:teuthology.orchestra.console.smithi050:Power off completed 2024-03-28T12:44:33.944 INFO:teuthology.lock.ops:unlocked: smithi050.front.sepia.ceph.com 2024-03-28T12:44:37.724 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T12:44:37.755 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T12:44:37.835 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is off 2024-03-28T12:44:37.865 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is off 2024-03-28T12:44:37.937 INFO:teuthology.orchestra.console.smithi163:Power off completed 2024-03-28T12:44:37.967 INFO:teuthology.orchestra.console.smithi169:Power off completed 2024-03-28T12:44:38.085 INFO:teuthology.lock.ops:unlocked: smithi163.front.sepia.ceph.com 2024-03-28T12:44:38.110 INFO:teuthology.lock.ops:unlocked: smithi169.front.sepia.ceph.com 2024-03-28T12:44:38.521 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 58, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 326, in reimage_machines reimaged = do_update_keys(list(reimaged.keys()))[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 290, in do_update_keys keys_dict = misc.ssh_keyscan(machines, _raise=_raise) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1108, in ssh_keyscan key = _ssh_keyscan(hostname) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1145, in _ssh_keyscan return sorted(keys)[0] IndexError: list index out of range