2024-03-18T14:31:29.834 INFO:root:teuthology version: 0.0.1.dev262+ge691533
2024-03-18T14:31:29.834 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-18T14:31:30.002 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7fb9569fef40>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7fb957a5fc10>, signals=[15])
2024-03-18T14:31:30.002 INFO:teuthology.lock.ops:Start node 'smithi032.front.sepia.ceph.com' reimaging
2024-03-18T14:31:30.002 INFO:teuthology.lock.ops:Updating [smithi032.front.sepia.ceph.com]: reset os type and version on server
2024-03-18T14:31:30.002 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server
2024-03-18T14:31:30.037 INFO:teuthology.lock.ops:Start node 'smithi066.front.sepia.ceph.com' reimaging
2024-03-18T14:31:30.038 INFO:teuthology.lock.ops:Updating [smithi066.front.sepia.ceph.com]: reset os type and version on server
2024-03-18T14:31:30.038 INFO:teuthology.lock.ops:Updating smithi066.front.sepia.ceph.com on lock server
2024-03-18T14:31:30.062 INFO:teuthology.lock.ops:Start node 'smithi203.front.sepia.ceph.com' reimaging
2024-03-18T14:31:30.062 INFO:teuthology.lock.ops:Updating [smithi203.front.sepia.ceph.com]: reset os type and version on server
2024-03-18T14:31:30.062 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server
2024-03-18T14:31:30.152 DEBUG:teuthology.provision.fog.smithi032:Requesting image smithi_centos_9.stream (ID 10)
2024-03-18T14:31:30.185 DEBUG:teuthology.provision.fog.smithi066:Requesting image smithi_centos_9.stream (ID 10)
2024-03-18T14:31:30.197 INFO:teuthology.provision.fog.smithi032:Scheduling deploy of centos 9.stream
2024-03-18T14:31:30.208 DEBUG:teuthology.provision.fog.smithi203:Requesting image smithi_centos_9.stream (ID 10)
2024-03-18T14:31:30.231 INFO:teuthology.provision.fog.smithi066:Scheduling deploy of centos 9.stream
2024-03-18T14:31:30.243 INFO:teuthology.provision.fog.smithi203:Scheduling deploy of centos 9.stream
2024-03-18T14:31:30.419 INFO:teuthology.orchestra.console.smithi032:Power off
2024-03-18T14:31:30.420 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-18T14:31:30.443 DEBUG:teuthology.orchestra.console.smithi032:power off output: Chassis Power Control: Down/Off
2024-03-18T14:31:30.444 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T14:31:30.458 INFO:teuthology.orchestra.console.smithi203:Power off
2024-03-18T14:31:30.458 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-18T14:31:30.470 INFO:teuthology.orchestra.console.smithi066:Power off
2024-03-18T14:31:30.470 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-18T14:31:30.477 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is off
2024-03-18T14:31:30.478 DEBUG:teuthology.orchestra.console.smithi203:power off output: Chassis Power Control: Down/Off
2024-03-18T14:31:30.479 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T14:31:30.491 DEBUG:teuthology.orchestra.console.smithi066:power off output: Chassis Power Control: Down/Off
2024-03-18T14:31:30.491 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T14:31:30.499 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off
2024-03-18T14:31:30.510 DEBUG:teuthology.orchestra.console.smithi066:check power output: Chassis Power is off
2024-03-18T14:31:30.578 INFO:teuthology.orchestra.console.smithi032:Power off completed
2024-03-18T14:31:30.600 INFO:teuthology.orchestra.console.smithi203:Power off completed
2024-03-18T14:31:30.611 INFO:teuthology.orchestra.console.smithi066:Power off completed
2024-03-18T14:31:30.679 INFO:teuthology.orchestra.console.smithi032:Power on
2024-03-18T14:31:30.679 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-03-18T14:31:30.701 DEBUG:teuthology.orchestra.console.smithi032:power on output: Chassis Power Control: Up/On
2024-03-18T14:31:30.701 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T14:31:30.707 INFO:teuthology.orchestra.console.smithi203:Power on
2024-03-18T14:31:30.708 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-03-18T14:31:30.714 INFO:teuthology.orchestra.console.smithi066:Power on
2024-03-18T14:31:30.714 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-03-18T14:31:30.724 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is off
2024-03-18T14:31:30.728 DEBUG:teuthology.orchestra.console.smithi203:power on output: Chassis Power Control: Up/On
2024-03-18T14:31:30.728 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T14:31:30.735 DEBUG:teuthology.orchestra.console.smithi066:power on output: Chassis Power Control: Up/On
2024-03-18T14:31:30.736 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T14:31:30.748 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off
2024-03-18T14:31:30.756 DEBUG:teuthology.orchestra.console.smithi066:check power output: Chassis Power is off
2024-03-18T14:31:34.729 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T14:31:34.750 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T14:31:34.756 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T14:31:34.839 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is off
2024-03-18T14:31:34.858 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off
2024-03-18T14:31:34.865 DEBUG:teuthology.orchestra.console.smithi066:check power output: Chassis Power is off
2024-03-18T14:31:38.842 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T14:31:38.859 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T14:31:38.866 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T14:31:38.952 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is on
2024-03-18T14:31:38.968 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on
2024-03-18T14:31:38.974 DEBUG:teuthology.orchestra.console.smithi066:check power output: Chassis Power is on
2024-03-18T14:31:39.053 INFO:teuthology.orchestra.console.smithi032:Power on completed
2024-03-18T14:31:39.068 INFO:teuthology.orchestra.console.smithi203:Power on completed
2024-03-18T14:31:39.076 INFO:teuthology.orchestra.console.smithi066:Power on completed
2024-03-18T14:31:39.154 INFO:teuthology.provision.fog.smithi032:Waiting for deploy to finish
2024-03-18T14:31:39.169 INFO:teuthology.provision.fog.smithi203:Waiting for deploy to finish
2024-03-18T14:31:39.177 INFO:teuthology.provision.fog.smithi066:Waiting for deploy to finish
2024-03-18T14:33:54.953 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T14:34:12.899 WARNING:teuthology.provision.fog.smithi032:[Errno None] Unable to connect to port 22 on 172.21.15.32
2024-03-18T14:34:18.903 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T14:34:21.986 WARNING:teuthology.provision.fog.smithi032:[Errno None] Unable to connect to port 22 on 172.21.15.32
2024-03-18T14:34:25.090 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T14:34:27.990 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T14:34:28.158 WARNING:teuthology.provision.fog.smithi203:[Errno None] Unable to connect to port 22 on 172.21.15.203
2024-03-18T14:34:31.070 WARNING:teuthology.provision.fog.smithi032:[Errno None] Unable to connect to port 22 on 172.21.15.32
2024-03-18T14:34:34.163 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T14:34:37.074 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T14:34:37.215 WARNING:teuthology.provision.fog.smithi203:[Errno None] Unable to connect to port 22 on 172.21.15.203
2024-03-18T14:34:43.219 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T14:34:46.274 WARNING:teuthology.provision.fog.smithi203:[Errno None] Unable to connect to port 22 on 172.21.15.203
2024-03-18T14:34:52.279 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T14:34:55.192 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T14:34:55.309 ERROR:teuthology.orchestra.connection:Error authenticating with smithi066.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-03-18T14:34:55.358 WARNING:teuthology.provision.fog.smithi203:[Errno None] Unable to connect to port 22 on 172.21.15.203
2024-03-18T14:35:01.360 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T14:35:04.414 WARNING:teuthology.provision.fog.smithi203:[Errno None] Unable to connect to port 22 on 172.21.15.203
2024-03-18T14:35:10.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T14:35:13.470 WARNING:teuthology.provision.fog.smithi203:[Errno None] Unable to connect to port 22 on 172.21.15.203
2024-03-18T14:35:19.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T14:35:37.077 WARNING:teuthology.provision.fog.smithi032:timed out
2024-03-18T14:35:43.079 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T14:35:43.401 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-03-18T14:35:43.402 DEBUG:teuthology.orchestra.run.smithi032:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-03-18T14:35:43.739 INFO:teuthology.provision.fog.smithi032:Node is ready
2024-03-18T14:35:43.739 DEBUG:teuthology.orchestra.run.smithi032:> hostname
2024-03-18T14:35:43.756 INFO:teuthology.orchestra.run.smithi032.stdout:smithi032.front.sepia.ceph.com
2024-03-18T14:35:43.756 DEBUG:teuthology.orchestra.run.smithi032:> grep smithi032.front.sepia.ceph.com /etc/hosts
2024-03-18T14:35:43.812 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32 smithi032.front.sepia.ceph.com smithi032
2024-03-18T14:35:43.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo hostname smithi032
2024-03-18T14:35:43.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i -e 's/smithi032.front.sepia.ceph.com/smithi032/g' /etc/hosts
2024-03-18T14:35:43.968 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i -e 's/172.21.15.32/172.21.15.32/g' /etc/hosts
2024-03-18T14:35:44.040 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i -e 's/smithi032.front.sepia.ceph.com/smithi032/g' /etc/hostname
2024-03-18T14:35:44.112 DEBUG:teuthology.orchestra.run.smithi032:> sudo hostname smithi032
2024-03-18T14:35:44.183 DEBUG:teuthology.orchestra.run.smithi032:> cat /etc/os-release
2024-03-18T14:35:44.239 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="CentOS Stream"
2024-03-18T14:35:44.239 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="9"
2024-03-18T14:35:44.244 INFO:teuthology.orchestra.run.smithi032.stdout:ID="centos"
2024-03-18T14:35:44.244 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE="rhel fedora"
2024-03-18T14:35:44.244 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="9"
2024-03-18T14:35:44.244 INFO:teuthology.orchestra.run.smithi032.stdout:PLATFORM_ID="platform:el9"
2024-03-18T14:35:44.244 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="CentOS Stream 9"
2024-03-18T14:35:44.245 INFO:teuthology.orchestra.run.smithi032.stdout:ANSI_COLOR="0;31"
2024-03-18T14:35:44.245 INFO:teuthology.orchestra.run.smithi032.stdout:LOGO="fedora-logo-icon"
2024-03-18T14:35:44.245 INFO:teuthology.orchestra.run.smithi032.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-03-18T14:35:44.245 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://centos.org/"
2024-03-18T14:35:44.245 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-03-18T14:35:44.245 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-03-18T14:35:44.245 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-03-18T14:35:44.246 INFO:teuthology.provision.fog.smithi032:Deploy complete!
2024-03-18T14:35:44.247 DEBUG:teuthology.parallel:result is None
2024-03-18T14:35:59.314 WARNING:teuthology.provision.fog.smithi066:timed out
2024-03-18T14:36:05.317 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T14:36:08.386 WARNING:teuthology.provision.fog.smithi066:[Errno None] Unable to connect to port 22 on 172.21.15.66
2024-03-18T14:36:14.392 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T14:36:17.470 WARNING:teuthology.provision.fog.smithi066:[Errno None] Unable to connect to port 22 on 172.21.15.66
2024-03-18T14:36:19.474 WARNING:teuthology.provision.fog.smithi203:timed out
2024-03-18T14:36:23.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T14:36:25.476 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T14:36:25.763 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-03-18T14:36:25.763 DEBUG:teuthology.orchestra.run.smithi203:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-03-18T14:36:26.109 INFO:teuthology.provision.fog.smithi203:Node is ready
2024-03-18T14:36:26.110 DEBUG:teuthology.orchestra.run.smithi203:> hostname
2024-03-18T14:36:26.127 INFO:teuthology.orchestra.run.smithi203.stdout:smithi203.front.sepia.ceph.com
2024-03-18T14:36:26.128 DEBUG:teuthology.orchestra.run.smithi203:> grep smithi203.front.sepia.ceph.com /etc/hosts
2024-03-18T14:36:26.182 INFO:teuthology.orchestra.run.smithi203.stdout:172.21.15.203 smithi203.front.sepia.ceph.com smithi203
2024-03-18T14:36:26.183 DEBUG:teuthology.orchestra.run.smithi203:> sudo hostname smithi203
2024-03-18T14:36:26.285 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i -e 's/smithi203.front.sepia.ceph.com/smithi203/g' /etc/hosts
2024-03-18T14:36:26.313 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i -e 's/172.21.15.203/172.21.15.203/g' /etc/hosts
2024-03-18T14:36:26.381 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i -e 's/smithi203.front.sepia.ceph.com/smithi203/g' /etc/hostname
2024-03-18T14:36:26.449 DEBUG:teuthology.orchestra.run.smithi203:> sudo hostname smithi203
2024-03-18T14:36:26.515 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release
2024-03-18T14:36:26.526 WARNING:teuthology.provision.fog.smithi066:[Errno None] Unable to connect to port 22 on 172.21.15.66
2024-03-18T14:36:26.570 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="CentOS Stream"
2024-03-18T14:36:26.570 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="9"
2024-03-18T14:36:26.570 INFO:teuthology.orchestra.run.smithi203.stdout:ID="centos"
2024-03-18T14:36:26.570 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE="rhel fedora"
2024-03-18T14:36:26.570 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="9"
2024-03-18T14:36:26.570 INFO:teuthology.orchestra.run.smithi203.stdout:PLATFORM_ID="platform:el9"
2024-03-18T14:36:26.570 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="CentOS Stream 9"
2024-03-18T14:36:26.571 INFO:teuthology.orchestra.run.smithi203.stdout:ANSI_COLOR="0;31"
2024-03-18T14:36:26.571 INFO:teuthology.orchestra.run.smithi203.stdout:LOGO="fedora-logo-icon"
2024-03-18T14:36:26.571 INFO:teuthology.orchestra.run.smithi203.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-03-18T14:36:26.571 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://centos.org/"
2024-03-18T14:36:26.571 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-03-18T14:36:26.571 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-03-18T14:36:26.571 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-03-18T14:36:26.571 INFO:teuthology.provision.fog.smithi203:Deploy complete!
2024-03-18T14:36:26.572 DEBUG:teuthology.parallel:result is None
2024-03-18T14:36:32.527 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T14:36:35.582 WARNING:teuthology.provision.fog.smithi066:[Errno None] Unable to connect to port 22 on 172.21.15.66
2024-03-18T14:36:41.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T14:37:41.586 WARNING:teuthology.provision.fog.smithi066:timed out
2024-03-18T14:37:47.589 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T14:37:47.867 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-03-18T14:37:47.867 DEBUG:teuthology.orchestra.run.smithi066:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-03-18T14:37:48.216 INFO:teuthology.provision.fog.smithi066:Node is ready
2024-03-18T14:37:48.216 DEBUG:teuthology.orchestra.run.smithi066:> hostname
2024-03-18T14:37:48.233 INFO:teuthology.orchestra.run.smithi066.stdout:smithi066.front.sepia.ceph.com
2024-03-18T14:37:48.233 DEBUG:teuthology.orchestra.run.smithi066:> grep smithi066.front.sepia.ceph.com /etc/hosts
2024-03-18T14:37:48.289 INFO:teuthology.orchestra.run.smithi066.stdout:172.21.15.66 smithi066.front.sepia.ceph.com smithi066
2024-03-18T14:37:48.289 DEBUG:teuthology.orchestra.run.smithi066:> sudo hostname smithi066
2024-03-18T14:37:48.398 DEBUG:teuthology.orchestra.run.smithi066:> sudo sed -i -e 's/smithi066.front.sepia.ceph.com/smithi066/g' /etc/hosts
2024-03-18T14:37:48.430 DEBUG:teuthology.orchestra.run.smithi066:> sudo sed -i -e 's/172.21.15.66/172.21.15.66/g' /etc/hosts
2024-03-18T14:37:48.501 DEBUG:teuthology.orchestra.run.smithi066:> sudo sed -i -e 's/smithi066.front.sepia.ceph.com/smithi066/g' /etc/hostname
2024-03-18T14:37:48.575 DEBUG:teuthology.orchestra.run.smithi066:> sudo hostname smithi066
2024-03-18T14:37:48.646 DEBUG:teuthology.orchestra.run.smithi066:> cat /etc/os-release
2024-03-18T14:37:48.701 INFO:teuthology.orchestra.run.smithi066.stdout:NAME="CentOS Stream"
2024-03-18T14:37:48.701 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION="9"
2024-03-18T14:37:48.701 INFO:teuthology.orchestra.run.smithi066.stdout:ID="centos"
2024-03-18T14:37:48.701 INFO:teuthology.orchestra.run.smithi066.stdout:ID_LIKE="rhel fedora"
2024-03-18T14:37:48.701 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION_ID="9"
2024-03-18T14:37:48.702 INFO:teuthology.orchestra.run.smithi066.stdout:PLATFORM_ID="platform:el9"
2024-03-18T14:37:48.702 INFO:teuthology.orchestra.run.smithi066.stdout:PRETTY_NAME="CentOS Stream 9"
2024-03-18T14:37:48.702 INFO:teuthology.orchestra.run.smithi066.stdout:ANSI_COLOR="0;31"
2024-03-18T14:37:48.702 INFO:teuthology.orchestra.run.smithi066.stdout:LOGO="fedora-logo-icon"
2024-03-18T14:37:48.702 INFO:teuthology.orchestra.run.smithi066.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-03-18T14:37:48.702 INFO:teuthology.orchestra.run.smithi066.stdout:HOME_URL="https://centos.org/"
2024-03-18T14:37:48.702 INFO:teuthology.orchestra.run.smithi066.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-03-18T14:37:48.702 INFO:teuthology.orchestra.run.smithi066.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-03-18T14:37:48.702 INFO:teuthology.orchestra.run.smithi066.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-03-18T14:37:48.702 INFO:teuthology.provision.fog.smithi066:Deploy complete!
2024-03-18T14:37:48.703 DEBUG:teuthology.parallel:result is None
2024-03-18T14:37:49.789 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 326, in reimage_machines
    reimaged = do_update_keys(list(reimaged.keys()))[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 290, in do_update_keys
    keys_dict = misc.ssh_keyscan(machines, _raise=_raise)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1108, in ssh_keyscan
    key = _ssh_keyscan(hostname)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1145, in _ssh_keyscan
    return sorted(keys)[0]
IndexError: list index out of range
2024-03-18T14:37:49.791 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-18T14:37:50.092 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ae895b5da806491ab92bc3545a0a6eee
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 326, in reimage_machines
    reimaged = do_update_keys(list(reimaged.keys()))[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 290, in do_update_keys
    keys_dict = misc.ssh_keyscan(machines, _raise=_raise)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1108, in ssh_keyscan
    key = _ssh_keyscan(hostname)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1145, in _ssh_keyscan
    return sorted(keys)[0]
IndexError: list index out of range
2024-03-18T14:37:50.094 INFO:teuthology.nuke:Checking targets against current locks
2024-03-18T14:37:50.157 DEBUG:teuthology.nuke:shortname: smithi032
2024-03-18T14:37:50.158 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-03-18T14:37:50.189 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608852', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 14:31:28.785738', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGVBEdWp/7bUqN/4Ta887X35k5f1TAs3uF7TSPYIKyuNV0FzYmeYE426Dj65U1Lo9x0km9WbUtaqfaDvHtxFa1Q='}
2024-03-18T14:37:50.190 DEBUG:teuthology.nuke:shortname: smithi066
2024-03-18T14:37:50.190 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-03-18T14:37:50.221 DEBUG:teuthology.nuke:shortname: smithi203
2024-03-18T14:37:50.221 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-03-18T14:37:50.228 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi066.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608852', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 14:31:28.784877', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKOupr+873jythyqddYjUuLgHkrAs79NF/9XdXedysS2IC4pmRjDEyMz6ic/VNK3zzITFMakMJXegqRfxaYaEDk='}
2024-03-18T14:37:50.249 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608852', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 14:31:28.783233', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJu+8THSYE5j1HCrNEi6trEJ3/q4yDHhntND70RLUBbzeFtZ1nlKe+tD0Ej13vf2KSpMfRyFuZgC8pfHmZIgNVM='}
2024-03-18T14:37:50.263 INFO:teuthology.orchestra.console.smithi032:Power off
2024-03-18T14:37:50.264 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-18T14:37:50.295 DEBUG:teuthology.orchestra.console.smithi032:power off output: Chassis Power Control: Down/Off
2024-03-18T14:37:50.295 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T14:37:50.324 INFO:teuthology.orchestra.console.smithi066:Power off
2024-03-18T14:37:50.325 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-18T14:37:50.339 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is off
2024-03-18T14:37:50.349 DEBUG:teuthology.orchestra.console.smithi066:power off output: Chassis Power Control: Down/Off
2024-03-18T14:37:50.349 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T14:37:50.373 INFO:teuthology.orchestra.console.smithi203:Power off
2024-03-18T14:37:50.373 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-18T14:37:50.386 DEBUG:teuthology.orchestra.console.smithi066:check power output: Chassis Power is on
2024-03-18T14:37:50.401 DEBUG:teuthology.orchestra.console.smithi203:power off output: Chassis Power Control: Down/Off
2024-03-18T14:37:50.401 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T14:37:50.426 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on
2024-03-18T14:37:50.439 INFO:teuthology.orchestra.console.smithi032:Power off completed
2024-03-18T14:37:50.588 INFO:teuthology.lock.ops:unlocked: smithi032.front.sepia.ceph.com
2024-03-18T14:37:54.388 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T14:37:54.427 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T14:37:54.498 DEBUG:teuthology.orchestra.console.smithi066:check power output: Chassis Power is on
2024-03-18T14:37:54.541 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on
2024-03-18T14:37:58.499 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T14:37:58.542 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T14:37:58.611 DEBUG:teuthology.orchestra.console.smithi066:check power output: Chassis Power is off
2024-03-18T14:37:58.653 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off
2024-03-18T14:37:58.712 INFO:teuthology.orchestra.console.smithi066:Power off completed
2024-03-18T14:37:58.753 INFO:teuthology.orchestra.console.smithi203:Power off completed
2024-03-18T14:37:58.871 INFO:teuthology.lock.ops:unlocked: smithi066.front.sepia.ceph.com
2024-03-18T14:37:58.904 INFO:teuthology.lock.ops:unlocked: smithi203.front.sepia.ceph.com
2024-03-18T14:37:59.320 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 326, in reimage_machines
    reimaged = do_update_keys(list(reimaged.keys()))[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 290, in do_update_keys
    keys_dict = misc.ssh_keyscan(machines, _raise=_raise)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1108, in ssh_keyscan
    key = _ssh_keyscan(hostname)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1145, in _ssh_keyscan
    return sorted(keys)[0]
IndexError: list index out of range