2024-03-25T08:10:44.045 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T08:10:44.046 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T08:10:44.194 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4843122c10>, signals=[15]) 2024-03-25T08:10:44.195 INFO:teuthology.lock.ops:Start node 'smithi183.front.sepia.ceph.com' reimaging 2024-03-25T08:10:44.195 INFO:teuthology.lock.ops:Updating [smithi183.front.sepia.ceph.com]: reset os type and version on server 2024-03-25T08:10:44.195 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2024-03-25T08:10:44.215 INFO:teuthology.lock.ops:Start node 'smithi184.front.sepia.ceph.com' reimaging 2024-03-25T08:10:44.218 INFO:teuthology.lock.ops:Updating [smithi184.front.sepia.ceph.com]: reset os type and version on server 2024-03-25T08:10:44.218 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2024-03-25T08:10:44.341 DEBUG:teuthology.provision.fog.smithi183:Requesting image smithi_centos_9.stream (ID 10) 2024-03-25T08:10:44.364 DEBUG:teuthology.provision.fog.smithi184:Requesting image smithi_centos_9.stream (ID 10) 2024-03-25T08:10:44.380 INFO:teuthology.provision.fog.smithi183:Scheduling deploy of centos 9.stream 2024-03-25T08:10:44.412 INFO:teuthology.provision.fog.smithi184:Scheduling deploy of centos 9.stream 2024-03-25T08:10:44.470 DEBUG:teuthology.provision.fog.smithi183:Canceling deploy task with ID 893025 2024-03-25T08:10:44.702 INFO:teuthology.orchestra.console.smithi184:Power off 2024-03-25T08:10:44.702 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T08:10:44.728 DEBUG:teuthology.orchestra.console.smithi184:power off output: Chassis Power Control: Down/Off 2024-03-25T08:10:44.729 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T08:10:44.744 INFO:teuthology.orchestra.console.smithi183:Power off 2024-03-25T08:10:44.744 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T08:10:44.753 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is off 2024-03-25T08:10:44.765 DEBUG:teuthology.orchestra.console.smithi183:power off output: Chassis Power Control: Down/Off 2024-03-25T08:10:44.765 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T08:10:44.785 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is off 2024-03-25T08:10:44.854 INFO:teuthology.orchestra.console.smithi184:Power off completed 2024-03-25T08:10:44.886 INFO:teuthology.orchestra.console.smithi183:Power off completed 2024-03-25T08:10:44.955 INFO:teuthology.orchestra.console.smithi184:Power on 2024-03-25T08:10:44.956 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-25T08:10:44.977 DEBUG:teuthology.orchestra.console.smithi184:power on output: Chassis Power Control: Up/On 2024-03-25T08:10:44.977 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T08:10:44.987 INFO:teuthology.orchestra.console.smithi183:Power on 2024-03-25T08:10:44.987 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-25T08:10:45.003 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is off 2024-03-25T08:10:45.010 DEBUG:teuthology.orchestra.console.smithi183:power on output: Chassis Power Control: Up/On 2024-03-25T08:10:45.010 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T08:10:45.032 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is off 2024-03-25T08:10:49.006 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T08:10:49.033 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T08:10:49.115 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is on 2024-03-25T08:10:49.141 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is off 2024-03-25T08:10:49.217 INFO:teuthology.orchestra.console.smithi184:Power on completed 2024-03-25T08:10:49.317 INFO:teuthology.provision.fog.smithi184:Waiting for deploy to finish 2024-03-25T08:10:53.144 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T08:10:53.254 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-25T08:10:53.355 INFO:teuthology.orchestra.console.smithi183:Power on completed 2024-03-25T08:10:53.455 INFO:teuthology.provision.fog.smithi183:Waiting for deploy to finish 2024-03-25T08:13:20.299 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T08:13:24.304 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T08:13:34.622 WARNING:teuthology.provision.fog.smithi183:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-03-25T08:13:40.624 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T08:13:42.622 WARNING:teuthology.provision.fog.smithi183:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-03-25T08:13:48.624 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T08:13:51.678 WARNING:teuthology.provision.fog.smithi183:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-03-25T08:13:57.680 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T08:14:00.738 WARNING:teuthology.provision.fog.smithi183:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-03-25T08:14:06.740 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T08:14:09.793 WARNING:teuthology.provision.fog.smithi183:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-03-25T08:14:15.795 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T08:14:18.846 WARNING:teuthology.provision.fog.smithi183:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-03-25T08:14:20.343 ERROR:teuthology.orchestra.connection:Error authenticating with smithi184.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 457, in connect server_key = t.get_remote_server_key() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 918, in get_remote_server_key raise SSHException("No existing session") paramiko.ssh_exception.SSHException: No existing session 2024-03-25T08:14:24.848 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T08:14:42.719 WARNING:teuthology.provision.fog.smithi184:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-03-25T08:14:43.230 WARNING:teuthology.provision.fog.smithi183:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-03-25T08:14:48.721 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T08:14:49.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T08:14:51.774 WARNING:teuthology.provision.fog.smithi184:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-03-25T08:14:52.286 WARNING:teuthology.provision.fog.smithi183:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-03-25T08:14:57.776 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T08:14:58.288 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T08:15:00.830 WARNING:teuthology.provision.fog.smithi184:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-03-25T08:15:01.342 WARNING:teuthology.provision.fog.smithi183:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-03-25T08:15:06.832 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T08:15:07.177 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-03-25T08:15:07.177 DEBUG:teuthology.orchestra.run.smithi184:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-03-25T08:15:07.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T08:15:07.590 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-03-25T08:15:07.591 DEBUG:teuthology.orchestra.run.smithi183:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-03-25T08:15:23.033 INFO:teuthology.provision.fog.smithi183:Node is ready 2024-03-25T08:15:23.034 DEBUG:teuthology.orchestra.run.smithi183:> hostname 2024-03-25T08:15:23.090 INFO:teuthology.orchestra.run.smithi183.stdout:smithi046 2024-03-25T08:15:23.091 DEBUG:teuthology.orchestra.run.smithi183:> grep smithi046 /etc/hosts 2024-03-25T08:15:23.147 INFO:teuthology.orchestra.run.smithi183.stdout:172.21.15.46 smithi046.front.sepia.ceph.com smithi046 2024-03-25T08:15:23.147 DEBUG:teuthology.orchestra.run.smithi183:> sudo hostname smithi183 2024-03-25T08:15:23.297 DEBUG:teuthology.orchestra.run.smithi183:> sudo sed -i -e 's/smithi046/smithi183/g' /etc/hosts 2024-03-25T08:15:23.326 DEBUG:teuthology.orchestra.run.smithi183:> sudo sed -i -e 's/172.21.15.46/172.21.15.183/g' /etc/hosts 2024-03-25T08:15:23.392 DEBUG:teuthology.orchestra.run.smithi183:> sudo sed -i -e 's/smithi046/smithi183/g' /etc/hostname 2024-03-25T08:15:23.422 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-03-25T08:15:23.459 DEBUG:teuthology.orchestra.run.smithi183:> sudo hostname smithi183 2024-03-25T08:15:23.525 DEBUG:teuthology.orchestra.run.smithi183:> cat /etc/os-release 2024-03-25T08:15:23.579 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="CentOS Stream" 2024-03-25T08:15:23.580 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="9" 2024-03-25T08:15:23.580 INFO:teuthology.orchestra.run.smithi183.stdout:ID="centos" 2024-03-25T08:15:23.580 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE="rhel fedora" 2024-03-25T08:15:23.580 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="9" 2024-03-25T08:15:23.580 INFO:teuthology.orchestra.run.smithi183.stdout:PLATFORM_ID="platform:el9" 2024-03-25T08:15:23.580 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T08:15:23.580 INFO:teuthology.orchestra.run.smithi183.stdout:ANSI_COLOR="0;31" 2024-03-25T08:15:23.580 INFO:teuthology.orchestra.run.smithi183.stdout:LOGO="fedora-logo-icon" 2024-03-25T08:15:23.580 INFO:teuthology.orchestra.run.smithi183.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T08:15:23.580 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://centos.org/" 2024-03-25T08:15:23.580 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T08:15:23.580 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T08:15:23.580 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T08:15:23.581 INFO:teuthology.provision.fog.smithi183:Deploy complete! 2024-03-25T08:15:23.582 DEBUG:teuthology.parallel:result is None 2024-03-25T08:15:27.525 INFO:teuthology.provision.fog.smithi184:Node is ready 2024-03-25T08:15:27.525 DEBUG:teuthology.orchestra.run.smithi184:> hostname 2024-03-25T08:15:27.543 INFO:teuthology.orchestra.run.smithi184.stdout:smithi184.front.sepia.ceph.com 2024-03-25T08:15:27.544 DEBUG:teuthology.orchestra.run.smithi184:> grep smithi184.front.sepia.ceph.com /etc/hosts 2024-03-25T08:15:27.601 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.15.184 smithi184.front.sepia.ceph.com smithi184 2024-03-25T08:15:27.602 DEBUG:teuthology.orchestra.run.smithi184:> sudo hostname smithi184 2024-03-25T08:15:27.731 DEBUG:teuthology.orchestra.run.smithi184:> sudo sed -i -e 's/smithi184.front.sepia.ceph.com/smithi184/g' /etc/hosts 2024-03-25T08:15:27.767 DEBUG:teuthology.orchestra.run.smithi184:> sudo sed -i -e 's/172.21.15.184/172.21.15.184/g' /etc/hosts 2024-03-25T08:15:27.840 DEBUG:teuthology.orchestra.run.smithi184:> sudo sed -i -e 's/smithi184.front.sepia.ceph.com/smithi184/g' /etc/hostname 2024-03-25T08:15:27.910 DEBUG:teuthology.orchestra.run.smithi184:> sudo hostname smithi184 2024-03-25T08:15:27.979 DEBUG:teuthology.orchestra.run.smithi184:> cat /etc/os-release 2024-03-25T08:15:28.034 INFO:teuthology.orchestra.run.smithi184.stdout:NAME="CentOS Stream" 2024-03-25T08:15:28.034 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION="9" 2024-03-25T08:15:28.034 INFO:teuthology.orchestra.run.smithi184.stdout:ID="centos" 2024-03-25T08:15:28.034 INFO:teuthology.orchestra.run.smithi184.stdout:ID_LIKE="rhel fedora" 2024-03-25T08:15:28.034 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_ID="9" 2024-03-25T08:15:28.035 INFO:teuthology.orchestra.run.smithi184.stdout:PLATFORM_ID="platform:el9" 2024-03-25T08:15:28.035 INFO:teuthology.orchestra.run.smithi184.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T08:15:28.035 INFO:teuthology.orchestra.run.smithi184.stdout:ANSI_COLOR="0;31" 2024-03-25T08:15:28.035 INFO:teuthology.orchestra.run.smithi184.stdout:LOGO="fedora-logo-icon" 2024-03-25T08:15:28.035 INFO:teuthology.orchestra.run.smithi184.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T08:15:28.035 INFO:teuthology.orchestra.run.smithi184.stdout:HOME_URL="https://centos.org/" 2024-03-25T08:15:28.035 INFO:teuthology.orchestra.run.smithi184.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T08:15:28.035 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T08:15:28.035 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T08:15:28.035 INFO:teuthology.provision.fog.smithi184:Deploy complete! 2024-03-25T08:15:28.036 DEBUG:teuthology.parallel:result is None 2024-03-25T08:15:28.127 ERROR:teuthology.misc:write (smithi183.front.sepia.ceph.com): Connection refused 2024-03-25T08:15:28.128 ERROR:teuthology.misc:write (smithi183.front.sepia.ceph.com): Connection refused 2024-03-25T08:15:28.128 ERROR:teuthology.misc:write (smithi183.front.sepia.ceph.com): Connection refused 2024-03-25T08:15:28.128 ERROR:teuthology.misc:write (smithi183.front.sepia.ceph.com): Connection refused 2024-03-25T08:15:28.128 ERROR:teuthology.misc:write (smithi183.front.sepia.ceph.com): Connection refused 2024-03-25T08:15:28.128 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 326, in reimage_machines reimaged = do_update_keys(list(reimaged.keys()))[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 290, in do_update_keys keys_dict = misc.ssh_keyscan(machines, _raise=_raise) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1108, in ssh_keyscan key = _ssh_keyscan(hostname) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1145, in _ssh_keyscan return sorted(keys)[0] IndexError: list index out of range 2024-03-25T08:15:28.130 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T08:15:28.431 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=503db3f52ae2412aacb3c33f19bf5f4f Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 326, in reimage_machines reimaged = do_update_keys(list(reimaged.keys()))[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 290, in do_update_keys keys_dict = misc.ssh_keyscan(machines, _raise=_raise) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1108, in ssh_keyscan key = _ssh_keyscan(hostname) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1145, in _ssh_keyscan return sorted(keys)[0] IndexError: list index out of range 2024-03-25T08:15:28.433 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T08:15:28.502 DEBUG:teuthology.nuke:shortname: smithi183 2024-03-25T08:15:28.502 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T08:15:28.523 DEBUG:teuthology.nuke:shortname: smithi184 2024-03-25T08:15:28.523 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T08:15:28.526 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620157', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 08:10:42.775072', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLEzFHiCwY7W1XRvvEU4SwXuseYHlzJzgGi470y2srJe5oyIasb1jnof9/O0d+tvY9B87OznWZeiTa/nI5qO3GQ='} 2024-03-25T08:15:28.546 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-24_21:24:02-fs-squid-distro-default-smithi/7620157', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 08:10:42.776488', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI6FyhsQuCkpTrth48NUH7zwtxxEjNeGEtDBWlvNdhbqSeUY1FjhAC/K3iiIkcx34kJvmmB8uALPE43+H5Hn3k4='} 2024-03-25T08:15:28.573 INFO:teuthology.orchestra.console.smithi183:Power off 2024-03-25T08:15:28.574 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T08:15:28.604 DEBUG:teuthology.orchestra.console.smithi183:power off output: Chassis Power Control: Down/Off 2024-03-25T08:15:28.604 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T08:15:28.621 INFO:teuthology.orchestra.console.smithi184:Power off 2024-03-25T08:15:28.621 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T08:15:28.629 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-25T08:15:28.643 DEBUG:teuthology.orchestra.console.smithi184:power off output: Chassis Power Control: Down/Off 2024-03-25T08:15:28.643 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T08:15:28.666 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is on 2024-03-25T08:15:32.630 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T08:15:32.667 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T08:15:32.741 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is off 2024-03-25T08:15:32.779 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is on 2024-03-25T08:15:32.842 INFO:teuthology.orchestra.console.smithi183:Power off completed 2024-03-25T08:15:32.992 INFO:teuthology.lock.ops:unlocked: smithi183.front.sepia.ceph.com 2024-03-25T08:15:36.784 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T08:15:36.893 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is on 2024-03-25T08:15:40.898 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T08:15:41.008 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is off 2024-03-25T08:15:41.109 INFO:teuthology.orchestra.console.smithi184:Power off completed 2024-03-25T08:15:41.258 INFO:teuthology.lock.ops:unlocked: smithi184.front.sepia.ceph.com 2024-03-25T08:15:41.501 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 58, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 326, in reimage_machines reimaged = do_update_keys(list(reimaged.keys()))[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 290, in do_update_keys keys_dict = misc.ssh_keyscan(machines, _raise=_raise) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1108, in ssh_keyscan key = _ssh_keyscan(hostname) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1145, in _ssh_keyscan return sorted(keys)[0] IndexError: list index out of range