2024-03-18T15:48:38.065 INFO:root:teuthology version: 0.0.1.dev262+ge691533
2024-03-18T15:48:38.066 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-18T15:48:38.229 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7fc1431dff70>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7fc144240c10>, signals=[15])
2024-03-18T15:48:38.230 INFO:teuthology.lock.ops:Start node 'smithi003.front.sepia.ceph.com' reimaging
2024-03-18T15:48:38.230 INFO:teuthology.lock.ops:Updating [smithi003.front.sepia.ceph.com]: reset os type and version on server
2024-03-18T15:48:38.230 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server
2024-03-18T15:48:38.249 INFO:teuthology.lock.ops:Start node 'smithi143.front.sepia.ceph.com' reimaging
2024-03-18T15:48:38.250 INFO:teuthology.lock.ops:Updating [smithi143.front.sepia.ceph.com]: reset os type and version on server
2024-03-18T15:48:38.250 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server
2024-03-18T15:48:38.273 INFO:teuthology.lock.ops:Start node 'smithi146.front.sepia.ceph.com' reimaging
2024-03-18T15:48:38.274 INFO:teuthology.lock.ops:Updating [smithi146.front.sepia.ceph.com]: reset os type and version on server
2024-03-18T15:48:38.274 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server
2024-03-18T15:48:38.368 DEBUG:teuthology.provision.fog.smithi003:Requesting image smithi_centos_9.stream (ID 10)
2024-03-18T15:48:38.386 DEBUG:teuthology.provision.fog.smithi143:Requesting image smithi_centos_9.stream (ID 10)
2024-03-18T15:48:38.407 DEBUG:teuthology.provision.fog.smithi146:Requesting image smithi_centos_9.stream (ID 10)
2024-03-18T15:48:38.422 INFO:teuthology.provision.fog.smithi003:Scheduling deploy of centos 9.stream
2024-03-18T15:48:38.430 INFO:teuthology.provision.fog.smithi143:Scheduling deploy of centos 9.stream
2024-03-18T15:48:38.450 INFO:teuthology.provision.fog.smithi146:Scheduling deploy of centos 9.stream
2024-03-18T15:48:38.489 DEBUG:teuthology.provision.fog.smithi143:Canceling deploy task with ID 870553
2024-03-18T15:48:38.695 INFO:teuthology.orchestra.console.smithi003:Power off
2024-03-18T15:48:38.695 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-18T15:48:38.707 INFO:teuthology.orchestra.console.smithi146:Power off
2024-03-18T15:48:38.708 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-18T15:48:38.717 DEBUG:teuthology.orchestra.console.smithi003:power off output: Chassis Power Control: Down/Off
2024-03-18T15:48:38.717 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T15:48:38.729 DEBUG:teuthology.orchestra.console.smithi146:power off output: Chassis Power Control: Down/Off
2024-03-18T15:48:38.729 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T15:48:38.738 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is off
2024-03-18T15:48:38.750 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is off
2024-03-18T15:48:38.822 INFO:teuthology.orchestra.console.smithi143:Power off
2024-03-18T15:48:38.822 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-18T15:48:38.839 INFO:teuthology.orchestra.console.smithi003:Power off completed
2024-03-18T15:48:38.842 DEBUG:teuthology.orchestra.console.smithi143:power off output: Chassis Power Control: Down/Off
2024-03-18T15:48:38.842 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T15:48:38.851 INFO:teuthology.orchestra.console.smithi146:Power off completed
2024-03-18T15:48:38.887 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is off
2024-03-18T15:48:38.940 INFO:teuthology.orchestra.console.smithi003:Power on
2024-03-18T15:48:38.940 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-03-18T15:48:38.953 INFO:teuthology.orchestra.console.smithi146:Power on
2024-03-18T15:48:38.953 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-03-18T15:48:38.962 DEBUG:teuthology.orchestra.console.smithi003:power on output: Chassis Power Control: Up/On
2024-03-18T15:48:38.962 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T15:48:38.975 DEBUG:teuthology.orchestra.console.smithi146:power on output: Chassis Power Control: Up/On
2024-03-18T15:48:38.975 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T15:48:38.984 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is off
2024-03-18T15:48:38.988 INFO:teuthology.orchestra.console.smithi143:Power off completed
2024-03-18T15:48:38.997 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is off
2024-03-18T15:48:39.089 INFO:teuthology.orchestra.console.smithi143:Power on
2024-03-18T15:48:39.089 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-03-18T15:48:39.111 DEBUG:teuthology.orchestra.console.smithi143:power on output: Chassis Power Control: Up/On
2024-03-18T15:48:39.112 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T15:48:39.134 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is off
2024-03-18T15:48:42.989 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T15:48:42.998 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T15:48:43.099 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is off
2024-03-18T15:48:43.106 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is off
2024-03-18T15:48:43.136 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T15:48:43.245 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is on
2024-03-18T15:48:43.345 INFO:teuthology.orchestra.console.smithi143:Power on completed
2024-03-18T15:48:43.446 INFO:teuthology.provision.fog.smithi143:Waiting for deploy to finish
2024-03-18T15:48:47.103 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T15:48:47.110 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T15:48:47.212 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on
2024-03-18T15:48:47.291 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on
2024-03-18T15:48:47.313 INFO:teuthology.orchestra.console.smithi003:Power on completed
2024-03-18T15:48:47.392 INFO:teuthology.orchestra.console.smithi146:Power on completed
2024-03-18T15:48:47.414 INFO:teuthology.provision.fog.smithi003:Waiting for deploy to finish
2024-03-18T15:48:47.494 INFO:teuthology.provision.fog.smithi146:Waiting for deploy to finish
2024-03-18T15:51:14.571 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T15:51:18.370 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T15:51:21.442 WARNING:teuthology.provision.fog.smithi003:[Errno None] Unable to connect to port 22 on 172.21.15.3
2024-03-18T15:51:27.449 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T15:51:30.526 WARNING:teuthology.provision.fog.smithi003:[Errno None] Unable to connect to port 22 on 172.21.15.3
2024-03-18T15:51:33.441 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T15:51:36.510 WARNING:teuthology.provision.fog.smithi146:[Errno None] Unable to connect to port 22 on 172.21.15.146
2024-03-18T15:51:36.527 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T15:51:39.582 WARNING:teuthology.provision.fog.smithi003:[Errno None] Unable to connect to port 22 on 172.21.15.3
2024-03-18T15:51:42.514 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T15:51:45.570 WARNING:teuthology.provision.fog.smithi146:[Errno None] Unable to connect to port 22 on 172.21.15.146
2024-03-18T15:51:45.583 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T15:51:48.638 WARNING:teuthology.provision.fog.smithi003:[Errno None] Unable to connect to port 22 on 172.21.15.3
2024-03-18T15:51:51.575 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T15:51:54.643 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T15:51:54.654 WARNING:teuthology.provision.fog.smithi146:[Errno None] Unable to connect to port 22 on 172.21.15.146
2024-03-18T15:51:57.694 WARNING:teuthology.provision.fog.smithi003:[Errno None] Unable to connect to port 22 on 172.21.15.3
2024-03-18T15:52:00.658 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T15:52:03.699 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T15:52:03.710 WARNING:teuthology.provision.fog.smithi146:[Errno None] Unable to connect to port 22 on 172.21.15.146
2024-03-18T15:52:09.712 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T15:52:12.766 WARNING:teuthology.provision.fog.smithi146:[Errno None] Unable to connect to port 22 on 172.21.15.146
2024-03-18T15:52:14.576 WARNING:teuthology.provision.fog.smithi143:timed out
2024-03-18T15:52:18.772 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T15:52:20.578 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T15:52:21.822 WARNING:teuthology.provision.fog.smithi146:[Errno None] Unable to connect to port 22 on 172.21.15.146
2024-03-18T15:52:27.827 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T15:52:38.942 WARNING:teuthology.provision.fog.smithi143:[Errno None] Unable to connect to port 22 on 172.21.15.143
2024-03-18T15:52:44.947 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T15:52:47.998 WARNING:teuthology.provision.fog.smithi143:[Errno None] Unable to connect to port 22 on 172.21.15.143
2024-03-18T15:52:54.003 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T15:52:54.281 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-03-18T15:52:54.281 DEBUG:teuthology.orchestra.run.smithi143:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-03-18T15:53:03.701 WARNING:teuthology.provision.fog.smithi003:timed out
2024-03-18T15:53:09.703 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T15:53:09.964 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-03-18T15:53:09.964 DEBUG:teuthology.orchestra.run.smithi003:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-03-18T15:53:10.288 INFO:teuthology.provision.fog.smithi003:Node is ready
2024-03-18T15:53:10.288 DEBUG:teuthology.orchestra.run.smithi003:> hostname
2024-03-18T15:53:10.303 INFO:teuthology.orchestra.run.smithi003.stdout:smithi003.front.sepia.ceph.com
2024-03-18T15:53:10.304 DEBUG:teuthology.orchestra.run.smithi003:> grep smithi003.front.sepia.ceph.com /etc/hosts
2024-03-18T15:53:10.359 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3 smithi003.front.sepia.ceph.com smithi003
2024-03-18T15:53:10.360 DEBUG:teuthology.orchestra.run.smithi003:> sudo hostname smithi003
2024-03-18T15:53:10.469 DEBUG:teuthology.orchestra.run.smithi003:> sudo sed -i -e 's/smithi003.front.sepia.ceph.com/smithi003/g' /etc/hosts
2024-03-18T15:53:10.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo sed -i -e 's/172.21.15.3/172.21.15.3/g' /etc/hosts
2024-03-18T15:53:10.571 DEBUG:teuthology.orchestra.run.smithi003:> sudo sed -i -e 's/smithi003.front.sepia.ceph.com/smithi003/g' /etc/hostname
2024-03-18T15:53:10.642 DEBUG:teuthology.orchestra.run.smithi003:> sudo hostname smithi003
2024-03-18T15:53:10.712 DEBUG:teuthology.orchestra.run.smithi003:> cat /etc/os-release
2024-03-18T15:53:10.767 INFO:teuthology.orchestra.run.smithi003.stdout:NAME="CentOS Stream"
2024-03-18T15:53:10.767 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION="9"
2024-03-18T15:53:10.768 INFO:teuthology.orchestra.run.smithi003.stdout:ID="centos"
2024-03-18T15:53:10.768 INFO:teuthology.orchestra.run.smithi003.stdout:ID_LIKE="rhel fedora"
2024-03-18T15:53:10.768 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_ID="9"
2024-03-18T15:53:10.768 INFO:teuthology.orchestra.run.smithi003.stdout:PLATFORM_ID="platform:el9"
2024-03-18T15:53:10.768 INFO:teuthology.orchestra.run.smithi003.stdout:PRETTY_NAME="CentOS Stream 9"
2024-03-18T15:53:10.768 INFO:teuthology.orchestra.run.smithi003.stdout:ANSI_COLOR="0;31"
2024-03-18T15:53:10.768 INFO:teuthology.orchestra.run.smithi003.stdout:LOGO="fedora-logo-icon"
2024-03-18T15:53:10.768 INFO:teuthology.orchestra.run.smithi003.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-03-18T15:53:10.768 INFO:teuthology.orchestra.run.smithi003.stdout:HOME_URL="https://centos.org/"
2024-03-18T15:53:10.768 INFO:teuthology.orchestra.run.smithi003.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-03-18T15:53:10.768 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-03-18T15:53:10.768 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-03-18T15:53:10.769 INFO:teuthology.provision.fog.smithi003:Deploy complete!
2024-03-18T15:53:10.770 DEBUG:teuthology.parallel:result is None
2024-03-18T15:53:14.611 INFO:teuthology.provision.fog.smithi143:Node is ready
2024-03-18T15:53:14.612 DEBUG:teuthology.orchestra.run.smithi143:> hostname
2024-03-18T15:53:14.630 INFO:teuthology.orchestra.run.smithi143.stdout:smithi143.front.sepia.ceph.com
2024-03-18T15:53:14.630 DEBUG:teuthology.orchestra.run.smithi143:> grep smithi143.front.sepia.ceph.com /etc/hosts
2024-03-18T15:53:14.688 INFO:teuthology.orchestra.run.smithi143.stdout:172.21.15.143 smithi143.front.sepia.ceph.com smithi143
2024-03-18T15:53:14.688 DEBUG:teuthology.orchestra.run.smithi143:> sudo hostname smithi143
2024-03-18T15:53:14.815 DEBUG:teuthology.orchestra.run.smithi143:> sudo sed -i -e 's/smithi143.front.sepia.ceph.com/smithi143/g' /etc/hosts
2024-03-18T15:53:14.846 DEBUG:teuthology.orchestra.run.smithi143:> sudo sed -i -e 's/172.21.15.143/172.21.15.143/g' /etc/hosts
2024-03-18T15:53:14.916 DEBUG:teuthology.orchestra.run.smithi143:> sudo sed -i -e 's/smithi143.front.sepia.ceph.com/smithi143/g' /etc/hostname
2024-03-18T15:53:14.985 DEBUG:teuthology.orchestra.run.smithi143:> sudo hostname smithi143
2024-03-18T15:53:15.053 DEBUG:teuthology.orchestra.run.smithi143:> cat /etc/os-release
2024-03-18T15:53:15.108 INFO:teuthology.orchestra.run.smithi143.stdout:NAME="CentOS Stream"
2024-03-18T15:53:15.108 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION="9"
2024-03-18T15:53:15.108 INFO:teuthology.orchestra.run.smithi143.stdout:ID="centos"
2024-03-18T15:53:15.108 INFO:teuthology.orchestra.run.smithi143.stdout:ID_LIKE="rhel fedora"
2024-03-18T15:53:15.109 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_ID="9"
2024-03-18T15:53:15.109 INFO:teuthology.orchestra.run.smithi143.stdout:PLATFORM_ID="platform:el9"
2024-03-18T15:53:15.109 INFO:teuthology.orchestra.run.smithi143.stdout:PRETTY_NAME="CentOS Stream 9"
2024-03-18T15:53:15.109 INFO:teuthology.orchestra.run.smithi143.stdout:ANSI_COLOR="0;31"
2024-03-18T15:53:15.109 INFO:teuthology.orchestra.run.smithi143.stdout:LOGO="fedora-logo-icon"
2024-03-18T15:53:15.109 INFO:teuthology.orchestra.run.smithi143.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-03-18T15:53:15.109 INFO:teuthology.orchestra.run.smithi143.stdout:HOME_URL="https://centos.org/"
2024-03-18T15:53:15.109 INFO:teuthology.orchestra.run.smithi143.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-03-18T15:53:15.109 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-03-18T15:53:15.109 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-03-18T15:53:15.109 INFO:teuthology.provision.fog.smithi143:Deploy complete!
2024-03-18T15:53:15.110 DEBUG:teuthology.parallel:result is None
2024-03-18T15:53:27.841 WARNING:teuthology.provision.fog.smithi146:timed out
2024-03-18T15:53:33.848 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-18T15:53:34.189 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-03-18T15:53:34.189 DEBUG:teuthology.orchestra.run.smithi146:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-03-18T15:53:34.520 INFO:teuthology.provision.fog.smithi146:Node is ready
2024-03-18T15:53:34.521 DEBUG:teuthology.orchestra.run.smithi146:> hostname
2024-03-18T15:53:34.535 INFO:teuthology.orchestra.run.smithi146.stdout:smithi146.front.sepia.ceph.com
2024-03-18T15:53:34.535 DEBUG:teuthology.orchestra.run.smithi146:> grep smithi146.front.sepia.ceph.com /etc/hosts
2024-03-18T15:53:34.590 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146 smithi146.front.sepia.ceph.com smithi146
2024-03-18T15:53:34.591 DEBUG:teuthology.orchestra.run.smithi146:> sudo hostname smithi146
2024-03-18T15:53:34.699 DEBUG:teuthology.orchestra.run.smithi146:> sudo sed -i -e 's/smithi146.front.sepia.ceph.com/smithi146/g' /etc/hosts
2024-03-18T15:53:34.726 DEBUG:teuthology.orchestra.run.smithi146:> sudo sed -i -e 's/172.21.15.146/172.21.15.146/g' /etc/hosts
2024-03-18T15:53:34.793 DEBUG:teuthology.orchestra.run.smithi146:> sudo sed -i -e 's/smithi146.front.sepia.ceph.com/smithi146/g' /etc/hostname
2024-03-18T15:53:34.860 DEBUG:teuthology.orchestra.run.smithi146:> sudo hostname smithi146
2024-03-18T15:53:34.926 DEBUG:teuthology.orchestra.run.smithi146:> cat /etc/os-release
2024-03-18T15:53:34.980 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="CentOS Stream"
2024-03-18T15:53:34.980 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="9"
2024-03-18T15:53:34.980 INFO:teuthology.orchestra.run.smithi146.stdout:ID="centos"
2024-03-18T15:53:34.981 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE="rhel fedora"
2024-03-18T15:53:34.981 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="9"
2024-03-18T15:53:34.981 INFO:teuthology.orchestra.run.smithi146.stdout:PLATFORM_ID="platform:el9"
2024-03-18T15:53:34.981 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="CentOS Stream 9"
2024-03-18T15:53:34.981 INFO:teuthology.orchestra.run.smithi146.stdout:ANSI_COLOR="0;31"
2024-03-18T15:53:34.981 INFO:teuthology.orchestra.run.smithi146.stdout:LOGO="fedora-logo-icon"
2024-03-18T15:53:34.981 INFO:teuthology.orchestra.run.smithi146.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-03-18T15:53:34.981 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://centos.org/"
2024-03-18T15:53:34.981 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-03-18T15:53:34.981 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-03-18T15:53:34.981 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-03-18T15:53:34.982 INFO:teuthology.provision.fog.smithi146:Deploy complete!
2024-03-18T15:53:34.982 DEBUG:teuthology.parallel:result is None
2024-03-18T15:53:36.122 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 326, in reimage_machines
    reimaged = do_update_keys(list(reimaged.keys()))[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 290, in do_update_keys
    keys_dict = misc.ssh_keyscan(machines, _raise=_raise)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1108, in ssh_keyscan
    key = _ssh_keyscan(hostname)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1145, in _ssh_keyscan
    return sorted(keys)[0]
IndexError: list index out of range
2024-03-18T15:53:36.124 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-18T15:53:36.404 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=8f105c7f01ce460596b1936c9a941c03
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 326, in reimage_machines
    reimaged = do_update_keys(list(reimaged.keys()))[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 290, in do_update_keys
    keys_dict = misc.ssh_keyscan(machines, _raise=_raise)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1108, in ssh_keyscan
    key = _ssh_keyscan(hostname)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1145, in _ssh_keyscan
    return sorted(keys)[0]
IndexError: list index out of range
2024-03-18T15:53:36.406 INFO:teuthology.nuke:Checking targets against current locks
2024-03-18T15:53:36.469 DEBUG:teuthology.nuke:shortname: smithi003
2024-03-18T15:53:36.470 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-03-18T15:53:36.473 INFO:teuthology.nuke:Not nuking smithi143.front.sepia.ceph.com because description doesn't match: pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi != None
2024-03-18T15:53:36.495 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608914', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 15:48:36.978182', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGT1MowzEkrWLOXyCIlz4AIRgBgLGxb+Qoa9onTD1isD54kX4MtFc2kS/XXXZHXXN1tZV2AiqZGFx8c+kItkn28='}
2024-03-18T15:53:36.519 DEBUG:teuthology.nuke:shortname: smithi146
2024-03-18T15:53:36.519 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-03-18T15:53:36.545 INFO:teuthology.orchestra.console.smithi003:Power off
2024-03-18T15:53:36.546 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-18T15:53:36.555 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608914', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 15:48:36.976634', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKWz7RPA2lRSMvBaGJfQ7+xVKb8S8dh2gyyE+NQskSWRvXvlq79eT+E/HT0njVngBSPgziG6sBUJYXY92yaMpwA='}
2024-03-18T15:53:36.569 DEBUG:teuthology.orchestra.console.smithi003:power off output: Chassis Power Control: Down/Off
2024-03-18T15:53:36.570 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T15:53:36.592 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on
2024-03-18T15:53:36.606 INFO:teuthology.orchestra.console.smithi146:Power off
2024-03-18T15:53:36.607 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-18T15:53:36.627 DEBUG:teuthology.orchestra.console.smithi146:power off output: Chassis Power Control: Down/Off
2024-03-18T15:53:36.628 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T15:53:36.650 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on
2024-03-18T15:53:40.596 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T15:53:40.651 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T15:53:40.706 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on
2024-03-18T15:53:40.798 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on
2024-03-18T15:53:44.711 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T15:53:44.799 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T15:53:44.820 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is off
2024-03-18T15:53:44.908 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on
2024-03-18T15:53:44.921 INFO:teuthology.orchestra.console.smithi003:Power off completed
2024-03-18T15:53:45.071 INFO:teuthology.lock.ops:unlocked: smithi003.front.sepia.ceph.com
2024-03-18T15:53:48.912 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-18T15:53:49.022 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is off
2024-03-18T15:53:49.123 INFO:teuthology.orchestra.console.smithi146:Power off completed
2024-03-18T15:53:49.271 INFO:teuthology.lock.ops:unlocked: smithi146.front.sepia.ceph.com
2024-03-18T15:53:49.272 ERROR:teuthology.nuke:Could not nuke the following targets:
targets:
  smithi143.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKYqBlWY6lsi+kc1ESHyf/UlF3yLRuu2CeScumLnt/2WWN1EONpMVA5b4pti7LblCrI1zy5v7LAEf5UngRCSQgs=
2024-03-18T15:53:49.739 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 326, in reimage_machines
    reimaged = do_update_keys(list(reimaged.keys()))[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 290, in do_update_keys
    keys_dict = misc.ssh_keyscan(machines, _raise=_raise)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1108, in ssh_keyscan
    key = _ssh_keyscan(hostname)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1145, in _ssh_keyscan
    return sorted(keys)[0]
IndexError: list index out of range