2024-04-18T23:09:57.095 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d
2024-04-18T23:09:57.095 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-04-18T23:09:57.247 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7fd7f0090f40>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7fd7f10f1c10>, signals=[15])
2024-04-18T23:09:57.248 INFO:teuthology.lock.ops:Start node 'smithi138.front.sepia.ceph.com' reimaging
2024-04-18T23:09:57.248 INFO:teuthology.lock.ops:Updating [smithi138.front.sepia.ceph.com]: reset os type and version on server
2024-04-18T23:09:57.248 INFO:teuthology.lock.ops:Updating smithi138.front.sepia.ceph.com on lock server
2024-04-18T23:09:57.275 INFO:teuthology.lock.ops:Start node 'smithi163.front.sepia.ceph.com' reimaging
2024-04-18T23:09:57.276 INFO:teuthology.lock.ops:Updating [smithi163.front.sepia.ceph.com]: reset os type and version on server
2024-04-18T23:09:57.276 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server
2024-04-18T23:09:57.292 INFO:teuthology.lock.ops:Start node 'smithi178.front.sepia.ceph.com' reimaging
2024-04-18T23:09:57.292 INFO:teuthology.lock.ops:Updating [smithi178.front.sepia.ceph.com]: reset os type and version on server
2024-04-18T23:09:57.292 INFO:teuthology.lock.ops:Updating smithi178.front.sepia.ceph.com on lock server
2024-04-18T23:09:57.386 DEBUG:teuthology.provision.fog.smithi138:Requesting image smithi_centos_8.stream (ID 3)
2024-04-18T23:09:57.394 DEBUG:teuthology.provision.fog.smithi163:Requesting image smithi_centos_8.stream (ID 3)
2024-04-18T23:09:57.422 DEBUG:teuthology.provision.fog.smithi178:Requesting image smithi_centos_8.stream (ID 3)
2024-04-18T23:09:57.425 INFO:teuthology.provision.fog.smithi138:Scheduling deploy of centos 8.stream
2024-04-18T23:09:57.439 INFO:teuthology.provision.fog.smithi163:Scheduling deploy of centos 8.stream
2024-04-18T23:09:57.463 INFO:teuthology.provision.fog.smithi178:Scheduling deploy of centos 8.stream
2024-04-18T23:09:57.679 INFO:teuthology.orchestra.console.smithi138:Power off
2024-04-18T23:09:57.679 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-18T23:09:57.694 INFO:teuthology.orchestra.console.smithi163:Power off
2024-04-18T23:09:57.695 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-18T23:09:57.706 DEBUG:teuthology.orchestra.console.smithi138:power off output: Chassis Power Control: Down/Off
2024-04-18T23:09:57.706 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T23:09:57.716 DEBUG:teuthology.orchestra.console.smithi163:power off output: Chassis Power Control: Down/Off
2024-04-18T23:09:57.716 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T23:09:57.731 INFO:teuthology.orchestra.console.smithi178:Power off
2024-04-18T23:09:57.731 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-18T23:09:57.739 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is off
2024-04-18T23:09:57.739 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is off
2024-04-18T23:09:57.751 DEBUG:teuthology.orchestra.console.smithi178:power off output: Chassis Power Control: Down/Off
2024-04-18T23:09:57.751 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T23:09:57.772 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is off
2024-04-18T23:09:57.840 INFO:teuthology.orchestra.console.smithi163:Power off completed
2024-04-18T23:09:57.840 INFO:teuthology.orchestra.console.smithi138:Power off completed
2024-04-18T23:09:57.874 INFO:teuthology.orchestra.console.smithi178:Power off completed
2024-04-18T23:09:57.941 INFO:teuthology.orchestra.console.smithi163:Power on
2024-04-18T23:09:57.941 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-04-18T23:09:57.949 INFO:teuthology.orchestra.console.smithi138:Power on
2024-04-18T23:09:57.950 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-04-18T23:09:57.966 DEBUG:teuthology.orchestra.console.smithi163:power on output: Chassis Power Control: Up/On
2024-04-18T23:09:57.966 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T23:09:57.974 DEBUG:teuthology.orchestra.console.smithi138:power on output: Chassis Power Control: Up/On
2024-04-18T23:09:57.975 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T23:09:57.981 INFO:teuthology.orchestra.console.smithi178:Power on
2024-04-18T23:09:57.981 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-04-18T23:09:57.990 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is off
2024-04-18T23:09:57.994 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is off
2024-04-18T23:09:58.002 DEBUG:teuthology.orchestra.console.smithi178:power on output: Chassis Power Control: Up/On
2024-04-18T23:09:58.002 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T23:09:58.022 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is off
2024-04-18T23:10:01.995 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T23:10:02.001 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T23:10:02.024 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T23:10:02.103 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is off
2024-04-18T23:10:02.109 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is off
2024-04-18T23:10:02.131 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is off
2024-04-18T23:10:06.106 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T23:10:06.113 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T23:10:06.132 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T23:10:06.215 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on
2024-04-18T23:10:06.221 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is on
2024-04-18T23:10:06.240 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is on
2024-04-18T23:10:06.316 INFO:teuthology.orchestra.console.smithi163:Power on completed
2024-04-18T23:10:06.322 INFO:teuthology.orchestra.console.smithi138:Power on completed
2024-04-18T23:10:06.342 INFO:teuthology.orchestra.console.smithi178:Power on completed
2024-04-18T23:10:06.418 INFO:teuthology.provision.fog.smithi163:Waiting for deploy to finish
2024-04-18T23:10:06.423 INFO:teuthology.provision.fog.smithi138:Waiting for deploy to finish
2024-04-18T23:10:06.443 INFO:teuthology.provision.fog.smithi178:Waiting for deploy to finish
2024-04-18T23:12:52.353 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:12:55.426 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138
2024-04-18T23:13:01.434 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:13:04.510 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138
2024-04-18T23:13:07.421 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:13:07.425 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:13:10.494 WARNING:teuthology.provision.fog.smithi178:[Errno None] Unable to connect to port 22 on 172.21.15.178
2024-04-18T23:13:10.512 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:13:13.566 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138
2024-04-18T23:13:16.498 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:13:19.554 WARNING:teuthology.provision.fog.smithi178:[Errno None] Unable to connect to port 22 on 172.21.15.178
2024-04-18T23:13:19.567 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:13:22.622 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138
2024-04-18T23:13:25.557 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:13:25.790 WARNING:teuthology.provision.fog.smithi163:[Errno None] Unable to connect to port 22 on 172.21.15.163
2024-04-18T23:13:28.627 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:13:28.638 WARNING:teuthology.provision.fog.smithi178:[Errno None] Unable to connect to port 22 on 172.21.15.178
2024-04-18T23:13:31.678 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138
2024-04-18T23:13:31.791 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:13:34.642 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:13:34.846 WARNING:teuthology.provision.fog.smithi163:[Errno None] Unable to connect to port 22 on 172.21.15.163
2024-04-18T23:13:37.682 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:13:37.694 WARNING:teuthology.provision.fog.smithi178:[Errno None] Unable to connect to port 22 on 172.21.15.178
2024-04-18T23:13:40.734 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138
2024-04-18T23:13:40.848 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:13:43.698 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:13:43.906 WARNING:teuthology.provision.fog.smithi163:[Errno None] Unable to connect to port 22 on 172.21.15.163
2024-04-18T23:13:46.739 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:13:46.754 WARNING:teuthology.provision.fog.smithi178:[Errno None] Unable to connect to port 22 on 172.21.15.178
2024-04-18T23:13:49.790 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138
2024-04-18T23:13:49.908 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:13:52.759 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:13:52.962 WARNING:teuthology.provision.fog.smithi163:[Errno None] Unable to connect to port 22 on 172.21.15.163
2024-04-18T23:13:55.795 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:13:55.838 WARNING:teuthology.provision.fog.smithi178:[Errno None] Unable to connect to port 22 on 172.21.15.178
2024-04-18T23:13:58.850 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138
2024-04-18T23:13:58.964 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:14:01.843 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:14:02.018 WARNING:teuthology.provision.fog.smithi163:[Errno None] Unable to connect to port 22 on 172.21.15.163
2024-04-18T23:14:04.856 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:14:04.898 WARNING:teuthology.provision.fog.smithi178:[Errno None] Unable to connect to port 22 on 172.21.15.178
2024-04-18T23:14:07.938 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138
2024-04-18T23:14:08.020 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:14:10.902 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:14:11.074 WARNING:teuthology.provision.fog.smithi163:[Errno None] Unable to connect to port 22 on 172.21.15.163
2024-04-18T23:14:13.942 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:14:17.022 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138
2024-04-18T23:14:17.076 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:14:23.027 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:15:10.952 WARNING:teuthology.provision.fog.smithi178:timed out
2024-04-18T23:15:16.959 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:15:17.078 WARNING:teuthology.provision.fog.smithi163:timed out
2024-04-18T23:15:17.228 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-04-18T23:15:17.229 DEBUG:teuthology.orchestra.run.smithi178:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-04-18T23:15:17.475 INFO:teuthology.provision.fog.smithi178:Node is ready
2024-04-18T23:15:17.476 DEBUG:teuthology.orchestra.run.smithi178:> hostname
2024-04-18T23:15:17.535 INFO:teuthology.orchestra.run.smithi178.stdout:smithi178.front.sepia.ceph.com
2024-04-18T23:15:17.535 DEBUG:teuthology.orchestra.run.smithi178:> grep smithi178.front.sepia.ceph.com /etc/hosts
2024-04-18T23:15:17.595 INFO:teuthology.orchestra.run.smithi178.stdout:172.21.15.178 smithi178.front.sepia.ceph.com smithi178
2024-04-18T23:15:17.596 DEBUG:teuthology.orchestra.run.smithi178:> sudo hostname smithi178
2024-04-18T23:15:17.701 DEBUG:teuthology.orchestra.run.smithi178:> sudo sed -i -e 's/smithi178.front.sepia.ceph.com/smithi178/g' /etc/hosts
2024-04-18T23:15:17.733 DEBUG:teuthology.orchestra.run.smithi178:> sudo sed -i -e 's/172.21.15.178/172.21.15.178/g' /etc/hosts
2024-04-18T23:15:17.802 DEBUG:teuthology.orchestra.run.smithi178:> sudo sed -i -e 's/smithi178.front.sepia.ceph.com/smithi178/g' /etc/hostname
2024-04-18T23:15:17.871 DEBUG:teuthology.orchestra.run.smithi178:> sudo hostname smithi178
2024-04-18T23:15:17.938 DEBUG:teuthology.orchestra.run.smithi178:> cat /etc/os-release
2024-04-18T23:15:17.996 INFO:teuthology.orchestra.run.smithi178.stdout:NAME="CentOS Stream"
2024-04-18T23:15:17.996 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION="8"
2024-04-18T23:15:17.996 INFO:teuthology.orchestra.run.smithi178.stdout:ID="centos"
2024-04-18T23:15:17.996 INFO:teuthology.orchestra.run.smithi178.stdout:ID_LIKE="rhel fedora"
2024-04-18T23:15:17.996 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION_ID="8"
2024-04-18T23:15:17.996 INFO:teuthology.orchestra.run.smithi178.stdout:PLATFORM_ID="platform:el8"
2024-04-18T23:15:17.997 INFO:teuthology.orchestra.run.smithi178.stdout:PRETTY_NAME="CentOS Stream 8"
2024-04-18T23:15:17.997 INFO:teuthology.orchestra.run.smithi178.stdout:ANSI_COLOR="0;31"
2024-04-18T23:15:17.997 INFO:teuthology.orchestra.run.smithi178.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2024-04-18T23:15:17.997 INFO:teuthology.orchestra.run.smithi178.stdout:HOME_URL="https://centos.org/"
2024-04-18T23:15:17.997 INFO:teuthology.orchestra.run.smithi178.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-04-18T23:15:17.997 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2024-04-18T23:15:17.997 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-04-18T23:15:17.998 INFO:teuthology.provision.fog.smithi178:Deploy complete!
2024-04-18T23:15:17.999 DEBUG:teuthology.parallel:result is None
2024-04-18T23:15:23.030 WARNING:teuthology.provision.fog.smithi138:timed out
2024-04-18T23:15:23.079 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:15:23.345 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-04-18T23:15:23.346 DEBUG:teuthology.orchestra.run.smithi163:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-04-18T23:15:23.576 INFO:teuthology.provision.fog.smithi163:Node is ready
2024-04-18T23:15:23.577 DEBUG:teuthology.orchestra.run.smithi163:> hostname
2024-04-18T23:15:23.635 INFO:teuthology.orchestra.run.smithi163.stdout:smithi163.front.sepia.ceph.com
2024-04-18T23:15:23.636 DEBUG:teuthology.orchestra.run.smithi163:> grep smithi163.front.sepia.ceph.com /etc/hosts
2024-04-18T23:15:23.695 INFO:teuthology.orchestra.run.smithi163.stdout:172.21.15.163 smithi163.front.sepia.ceph.com smithi163
2024-04-18T23:15:23.695 DEBUG:teuthology.orchestra.run.smithi163:> sudo hostname smithi163
2024-04-18T23:15:23.807 DEBUG:teuthology.orchestra.run.smithi163:> sudo sed -i -e 's/smithi163.front.sepia.ceph.com/smithi163/g' /etc/hosts
2024-04-18T23:15:23.836 DEBUG:teuthology.orchestra.run.smithi163:> sudo sed -i -e 's/172.21.15.163/172.21.15.163/g' /etc/hosts
2024-04-18T23:15:23.904 DEBUG:teuthology.orchestra.run.smithi163:> sudo sed -i -e 's/smithi163.front.sepia.ceph.com/smithi163/g' /etc/hostname
2024-04-18T23:15:23.972 DEBUG:teuthology.orchestra.run.smithi163:> sudo hostname smithi163
2024-04-18T23:15:24.039 DEBUG:teuthology.orchestra.run.smithi163:> cat /etc/os-release
2024-04-18T23:15:24.098 INFO:teuthology.orchestra.run.smithi163.stdout:NAME="CentOS Stream"
2024-04-18T23:15:24.098 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION="8"
2024-04-18T23:15:24.098 INFO:teuthology.orchestra.run.smithi163.stdout:ID="centos"
2024-04-18T23:15:24.098 INFO:teuthology.orchestra.run.smithi163.stdout:ID_LIKE="rhel fedora"
2024-04-18T23:15:24.098 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION_ID="8"
2024-04-18T23:15:24.099 INFO:teuthology.orchestra.run.smithi163.stdout:PLATFORM_ID="platform:el8"
2024-04-18T23:15:24.099 INFO:teuthology.orchestra.run.smithi163.stdout:PRETTY_NAME="CentOS Stream 8"
2024-04-18T23:15:24.099 INFO:teuthology.orchestra.run.smithi163.stdout:ANSI_COLOR="0;31"
2024-04-18T23:15:24.099 INFO:teuthology.orchestra.run.smithi163.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2024-04-18T23:15:24.099 INFO:teuthology.orchestra.run.smithi163.stdout:HOME_URL="https://centos.org/"
2024-04-18T23:15:24.099 INFO:teuthology.orchestra.run.smithi163.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-04-18T23:15:24.099 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2024-04-18T23:15:24.099 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-04-18T23:15:24.099 INFO:teuthology.provision.fog.smithi163:Deploy complete!
2024-04-18T23:15:24.100 DEBUG:teuthology.parallel:result is None
2024-04-18T23:15:29.035 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-18T23:15:29.262 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-04-18T23:15:29.262 DEBUG:teuthology.orchestra.run.smithi138:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-04-18T23:15:29.509 INFO:teuthology.provision.fog.smithi138:Node is ready
2024-04-18T23:15:29.509 DEBUG:teuthology.orchestra.run.smithi138:> hostname
2024-04-18T23:15:29.529 INFO:teuthology.orchestra.run.smithi138.stdout:smithi138.front.sepia.ceph.com
2024-04-18T23:15:29.529 DEBUG:teuthology.orchestra.run.smithi138:> grep smithi138.front.sepia.ceph.com /etc/hosts
2024-04-18T23:15:29.589 INFO:teuthology.orchestra.run.smithi138.stdout:172.21.15.138 smithi138.front.sepia.ceph.com smithi138
2024-04-18T23:15:29.590 DEBUG:teuthology.orchestra.run.smithi138:> sudo hostname smithi138
2024-04-18T23:15:29.696 DEBUG:teuthology.orchestra.run.smithi138:> sudo sed -i -e 's/smithi138.front.sepia.ceph.com/smithi138/g' /etc/hosts
2024-04-18T23:15:29.727 DEBUG:teuthology.orchestra.run.smithi138:> sudo sed -i -e 's/172.21.15.138/172.21.15.138/g' /etc/hosts
2024-04-18T23:15:29.797 DEBUG:teuthology.orchestra.run.smithi138:> sudo sed -i -e 's/smithi138.front.sepia.ceph.com/smithi138/g' /etc/hostname
2024-04-18T23:15:29.864 DEBUG:teuthology.orchestra.run.smithi138:> sudo hostname smithi138
2024-04-18T23:15:29.931 DEBUG:teuthology.orchestra.run.smithi138:> cat /etc/os-release
2024-04-18T23:15:29.990 INFO:teuthology.orchestra.run.smithi138.stdout:NAME="CentOS Stream"
2024-04-18T23:15:29.991 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION="8"
2024-04-18T23:15:29.991 INFO:teuthology.orchestra.run.smithi138.stdout:ID="centos"
2024-04-18T23:15:29.991 INFO:teuthology.orchestra.run.smithi138.stdout:ID_LIKE="rhel fedora"
2024-04-18T23:15:29.991 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION_ID="8"
2024-04-18T23:15:29.991 INFO:teuthology.orchestra.run.smithi138.stdout:PLATFORM_ID="platform:el8"
2024-04-18T23:15:29.991 INFO:teuthology.orchestra.run.smithi138.stdout:PRETTY_NAME="CentOS Stream 8"
2024-04-18T23:15:29.991 INFO:teuthology.orchestra.run.smithi138.stdout:ANSI_COLOR="0;31"
2024-04-18T23:15:29.991 INFO:teuthology.orchestra.run.smithi138.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2024-04-18T23:15:29.991 INFO:teuthology.orchestra.run.smithi138.stdout:HOME_URL="https://centos.org/"
2024-04-18T23:15:29.991 INFO:teuthology.orchestra.run.smithi138.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-04-18T23:15:29.991 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2024-04-18T23:15:29.991 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-04-18T23:15:29.992 INFO:teuthology.provision.fog.smithi138:Deploy complete!
2024-04-18T23:15:29.992 DEBUG:teuthology.parallel:result is None
2024-04-18T23:15:30.131 ERROR:teuthology.misc:write (smithi163.front.sepia.ceph.com): Connection refused
2024-04-18T23:15:30.131 ERROR:teuthology.misc:write (smithi163.front.sepia.ceph.com): Connection refused
2024-04-18T23:15:30.131 ERROR:teuthology.misc:write (smithi163.front.sepia.ceph.com): Connection refused
2024-04-18T23:15:30.131 ERROR:teuthology.misc:write (smithi163.front.sepia.ceph.com): Connection refused
2024-04-18T23:15:30.131 ERROR:teuthology.misc:write (smithi163.front.sepia.ceph.com): Connection refused
2024-04-18T23:15:30.131 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/lock/ops.py", line 326, in reimage_machines
    reimaged = do_update_keys(list(reimaged.keys()))[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/lock/ops.py", line 290, in do_update_keys
    keys_dict = misc.ssh_keyscan(machines, _raise=_raise)
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/misc.py", line 1108, in ssh_keyscan
    key = _ssh_keyscan(hostname)
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/misc.py", line 1145, in _ssh_keyscan
    return sorted(keys)[0]
IndexError: list index out of range
2024-04-18T23:15:30.133 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-04-18T23:15:30.478 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=191a240359fb40849fdb5e3274709005
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/lock/ops.py", line 326, in reimage_machines
    reimaged = do_update_keys(list(reimaged.keys()))[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/lock/ops.py", line 290, in do_update_keys
    keys_dict = misc.ssh_keyscan(machines, _raise=_raise)
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/misc.py", line 1108, in ssh_keyscan
    key = _ssh_keyscan(hostname)
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/misc.py", line 1145, in _ssh_keyscan
    return sorted(keys)[0]
IndexError: list index out of range
2024-04-18T23:15:30.480 INFO:teuthology.nuke:Checking targets against current locks
2024-04-18T23:15:30.549 DEBUG:teuthology.nuke:shortname: smithi138
2024-04-18T23:15:30.550 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-04-18T23:15:30.580 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi138.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663029', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-18 23:09:56.019760', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP3gJ7BVPgQadsYL7B8lCtU1DOpIHUHBswoQKAFpnG+ZrDULmXvRgu0FkdFvmZdsc+e89yCUpC55ULhK7XEov6o='}
2024-04-18T23:15:30.584 DEBUG:teuthology.nuke:shortname: smithi163
2024-04-18T23:15:30.584 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-04-18T23:15:30.608 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663029', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-18 23:09:56.016808', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKQ94I4Fcm9N8ClFGQP8osahfVfKc0WfBGsclmDZv5WSMAywzADKNsqxxlIQmTD38MewvsDZlM4rSP1WdAF3V6M='}
2024-04-18T23:15:30.613 DEBUG:teuthology.nuke:shortname: smithi178
2024-04-18T23:15:30.613 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-04-18T23:15:30.641 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663029', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-18 23:09:56.018350', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNDS6yv531d8g4YzqzwT1Ketbv7LAWXNVrcMHSSKsv8k4d1HmXZRLZ0BsfpTYb5XVZ5I1AyEPJ4J0m4THfsvzUg='}
2024-04-18T23:15:30.658 INFO:teuthology.orchestra.console.smithi138:Power off
2024-04-18T23:15:30.660 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-18T23:15:30.688 DEBUG:teuthology.orchestra.console.smithi138:power off output: Chassis Power Control: Down/Off
2024-04-18T23:15:30.688 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T23:15:30.708 INFO:teuthology.orchestra.console.smithi163:Power off
2024-04-18T23:15:30.709 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-18T23:15:30.723 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is on
2024-04-18T23:15:30.735 DEBUG:teuthology.orchestra.console.smithi163:power off output: Chassis Power Control: Down/Off
2024-04-18T23:15:30.736 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T23:15:30.760 INFO:teuthology.orchestra.console.smithi178:Power off
2024-04-18T23:15:30.761 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-18T23:15:30.774 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on
2024-04-18T23:15:30.790 DEBUG:teuthology.orchestra.console.smithi178:power off output: Chassis Power Control: Down/Off
2024-04-18T23:15:30.790 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T23:15:30.815 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is on
2024-04-18T23:15:34.726 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T23:15:34.776 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T23:15:34.816 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T23:15:34.846 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is on
2024-04-18T23:15:34.887 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is off
2024-04-18T23:15:34.930 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is on
2024-04-18T23:15:34.990 INFO:teuthology.orchestra.console.smithi163:Power off completed
2024-04-18T23:15:35.141 INFO:teuthology.lock.ops:unlocked: smithi163.front.sepia.ceph.com
2024-04-18T23:15:38.850 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T23:15:38.931 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T23:15:38.964 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is on
2024-04-18T23:15:39.044 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is off
2024-04-18T23:15:39.144 INFO:teuthology.orchestra.console.smithi178:Power off completed
2024-04-18T23:15:39.300 INFO:teuthology.lock.ops:unlocked: smithi178.front.sepia.ceph.com
2024-04-18T23:15:42.967 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-18T23:15:43.078 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is off
2024-04-18T23:15:43.178 INFO:teuthology.orchestra.console.smithi138:Power off completed
2024-04-18T23:15:43.333 INFO:teuthology.lock.ops:unlocked: smithi138.front.sepia.ceph.com
2024-04-18T23:15:43.724 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/lock/ops.py", line 326, in reimage_machines
    reimaged = do_update_keys(list(reimaged.keys()))[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/lock/ops.py", line 290, in do_update_keys
    keys_dict = misc.ssh_keyscan(machines, _raise=_raise)
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/misc.py", line 1108, in ssh_keyscan
    key = _ssh_keyscan(hostname)
  File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/misc.py", line 1145, in _ssh_keyscan
    return sorted(keys)[0]
IndexError: list index out of range