2024-04-12T22:42:03.718 INFO:root:teuthology version: 0.0.1.dev267+g6c63784
2024-04-12T22:42:03.718 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-04-12T22:42:03.896 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7fd58b526f40>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7fd58c55dc10>, signals=[15])
2024-04-12T22:42:03.897 INFO:teuthology.lock.ops:Start node 'smithi026.front.sepia.ceph.com' reimaging
2024-04-12T22:42:03.897 INFO:teuthology.lock.ops:Updating [smithi026.front.sepia.ceph.com]: reset os type and version on server
2024-04-12T22:42:03.897 INFO:teuthology.lock.ops:Updating smithi026.front.sepia.ceph.com on lock server
2024-04-12T22:42:03.922 INFO:teuthology.lock.ops:Start node 'smithi120.front.sepia.ceph.com' reimaging
2024-04-12T22:42:03.922 INFO:teuthology.lock.ops:Updating [smithi120.front.sepia.ceph.com]: reset os type and version on server
2024-04-12T22:42:03.922 INFO:teuthology.lock.ops:Updating smithi120.front.sepia.ceph.com on lock server
2024-04-12T22:42:03.946 INFO:teuthology.lock.ops:Start node 'smithi156.front.sepia.ceph.com' reimaging
2024-04-12T22:42:03.947 INFO:teuthology.lock.ops:Updating [smithi156.front.sepia.ceph.com]: reset os type and version on server
2024-04-12T22:42:03.947 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server
2024-04-12T22:42:04.040 DEBUG:teuthology.provision.fog.smithi026:Requesting image smithi_centos_9.stream (ID 10)
2024-04-12T22:42:04.055 DEBUG:teuthology.provision.fog.smithi120:Requesting image smithi_centos_9.stream (ID 10)
2024-04-12T22:42:04.082 DEBUG:teuthology.provision.fog.smithi156:Requesting image smithi_centos_9.stream (ID 10)
2024-04-12T22:42:04.088 INFO:teuthology.provision.fog.smithi026:Scheduling deploy of centos 9.stream
2024-04-12T22:42:04.103 INFO:teuthology.provision.fog.smithi120:Scheduling deploy of centos 9.stream
2024-04-12T22:42:04.138 INFO:teuthology.provision.fog.smithi156:Scheduling deploy of centos 9.stream
2024-04-12T22:42:04.379 INFO:teuthology.orchestra.console.smithi120:Power off
2024-04-12T22:42:04.379 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-12T22:42:04.392 INFO:teuthology.orchestra.console.smithi026:Power off
2024-04-12T22:42:04.392 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-12T22:42:04.406 DEBUG:teuthology.orchestra.console.smithi120:power off output: Chassis Power Control: Down/Off
2024-04-12T22:42:04.406 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:42:04.415 DEBUG:teuthology.orchestra.console.smithi026:power off output: Chassis Power Control: Down/Off
2024-04-12T22:42:04.415 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:42:04.428 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is on
2024-04-12T22:42:04.440 INFO:teuthology.orchestra.console.smithi156:Power off
2024-04-12T22:42:04.440 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-12T22:42:04.449 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on
2024-04-12T22:42:04.460 DEBUG:teuthology.orchestra.console.smithi156:power off output: Chassis Power Control: Down/Off
2024-04-12T22:42:04.461 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:42:04.481 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is on
2024-04-12T22:42:08.431 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:42:08.450 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:42:08.482 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:42:08.540 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is on
2024-04-12T22:42:08.559 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on
2024-04-12T22:42:08.592 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off
2024-04-12T22:42:08.692 INFO:teuthology.orchestra.console.smithi156:Power off completed
2024-04-12T22:42:08.793 INFO:teuthology.orchestra.console.smithi156:Power on
2024-04-12T22:42:08.793 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-04-12T22:42:08.816 DEBUG:teuthology.orchestra.console.smithi156:power on output: Chassis Power Control: Up/On
2024-04-12T22:42:08.817 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:42:08.836 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off
2024-04-12T22:42:12.542 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:42:12.560 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:42:12.651 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is off
2024-04-12T22:42:12.671 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is off
2024-04-12T22:42:12.752 INFO:teuthology.orchestra.console.smithi120:Power off completed
2024-04-12T22:42:12.772 INFO:teuthology.orchestra.console.smithi026:Power off completed
2024-04-12T22:42:12.837 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:42:12.854 INFO:teuthology.orchestra.console.smithi120:Power on
2024-04-12T22:42:12.854 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-04-12T22:42:12.873 INFO:teuthology.orchestra.console.smithi026:Power on
2024-04-12T22:42:12.873 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-04-12T22:42:12.881 DEBUG:teuthology.orchestra.console.smithi120:power on output: Chassis Power Control: Up/On
2024-04-12T22:42:12.882 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:42:12.894 DEBUG:teuthology.orchestra.console.smithi026:power on output: Chassis Power Control: Up/On
2024-04-12T22:42:12.895 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:42:12.902 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is off
2024-04-12T22:42:12.915 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is off
2024-04-12T22:42:12.947 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off
2024-04-12T22:42:16.907 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:42:16.916 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:42:16.948 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:42:17.017 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is off
2024-04-12T22:42:17.024 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on
2024-04-12T22:42:17.057 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off
2024-04-12T22:42:17.125 INFO:teuthology.orchestra.console.smithi026:Power on completed
2024-04-12T22:42:17.225 INFO:teuthology.provision.fog.smithi026:Waiting for deploy to finish
2024-04-12T22:42:21.021 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:42:21.058 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:42:21.130 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is on
2024-04-12T22:42:21.168 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off
2024-04-12T22:42:21.232 INFO:teuthology.orchestra.console.smithi120:Power on completed
2024-04-12T22:42:21.332 INFO:teuthology.provision.fog.smithi120:Waiting for deploy to finish
2024-04-12T22:42:25.172 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:42:25.282 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off
2024-04-12T22:42:29.282 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:42:29.392 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off
2024-04-12T22:42:33.394 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:42:33.503 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off
2024-04-12T22:42:37.505 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:42:37.615 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off
2024-04-12T22:42:41.618 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:42:41.727 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off
2024-04-12T22:42:45.730 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:42:45.840 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off
2024-04-12T22:42:49.843 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:42:49.952 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off
2024-04-12T22:42:53.955 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:42:54.064 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off
2024-04-12T22:42:58.066 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:42:58.176 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off
2024-04-12T22:43:02.179 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:43:02.288 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off
2024-04-12T22:43:06.292 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:43:06.402 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off
2024-04-12T22:43:06.402 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2024-04-12T22:43:06.503 DEBUG:teuthology.provision.fog.smithi156:Canceling deploy task with ID 951129
2024-04-12T22:43:06.577 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi156
2024-04-12T22:43:06.579 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-04-12T22:43:06.860 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f60403db156d4896943a683bacf5d306
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi156
2024-04-12T22:43:06.862 INFO:teuthology.nuke:Checking targets against current locks
2024-04-12T22:43:06.924 DEBUG:teuthology.nuke:shortname: smithi026
2024-04-12T22:43:06.924 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-04-12T22:43:06.955 DEBUG:teuthology.nuke:shortname: smithi120
2024-04-12T22:43:06.955 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-04-12T22:43:06.956 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi026.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-12_21:08:02-orch-squid-distro-default-smithi/7654635', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 22:42:02.695178', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI22NooiFP7SvCwJ/5QFw7+EMlgU8mW5Yzlrds4h3YpxOvWVNDn7yFAzB7UCTJBKOHqUN2Q5JhK0GDiwFOqXRNY='}
2024-04-12T22:43:06.979 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi120.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-12_21:08:02-orch-squid-distro-default-smithi/7654635', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 22:42:02.698842', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNFdwFZuIR3rZtr4/ik2Zq1Q4HNRtuEnb5QSr4hHgFJZ2Vf2Ycmu8i4RVvPhnJl5XqNhQ5MamqaTVJNU//zEHA4='}
2024-04-12T22:43:06.985 DEBUG:teuthology.nuke:shortname: smithi156
2024-04-12T22:43:06.986 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-04-12T22:43:07.010 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-12_21:08:02-orch-squid-distro-default-smithi/7654635', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 22:42:02.697110', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIkLpB0iC9zbUOQE9/Kcu+kh3JZtBhvTclnHRrtDbj4oYPdaD2bglXPfKPYcdO7e/Gf5cuhYFZ12DaMPgn/xMao='}
2024-04-12T22:43:07.017 INFO:teuthology.orchestra.console.smithi026:Power off
2024-04-12T22:43:07.017 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-12T22:43:07.039 DEBUG:teuthology.orchestra.console.smithi026:power off output: Chassis Power Control: Down/Off
2024-04-12T22:43:07.040 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:43:07.056 INFO:teuthology.orchestra.console.smithi120:Power off
2024-04-12T22:43:07.056 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-12T22:43:07.069 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on
2024-04-12T22:43:07.077 DEBUG:teuthology.orchestra.console.smithi120:power off output: Chassis Power Control: Down/Off
2024-04-12T22:43:07.077 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:43:07.098 INFO:teuthology.orchestra.console.smithi156:Power off
2024-04-12T22:43:07.098 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-12T22:43:07.107 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is on
2024-04-12T22:43:07.119 DEBUG:teuthology.orchestra.console.smithi156:power off output: Chassis Power Control: Down/Off
2024-04-12T22:43:07.120 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:43:07.142 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off
2024-04-12T22:43:07.243 INFO:teuthology.orchestra.console.smithi156:Power off completed
2024-04-12T22:43:07.389 INFO:teuthology.lock.ops:unlocked: smithi156.front.sepia.ceph.com
2024-04-12T22:43:11.074 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:43:11.108 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-12T22:43:11.183 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is off
2024-04-12T22:43:11.216 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is off
2024-04-12T22:43:11.285 INFO:teuthology.orchestra.console.smithi026:Power off completed
2024-04-12T22:43:11.317 INFO:teuthology.orchestra.console.smithi120:Power off completed
2024-04-12T22:43:11.439 INFO:teuthology.lock.ops:unlocked: smithi026.front.sepia.ceph.com
2024-04-12T22:43:11.464 INFO:teuthology.lock.ops:unlocked: smithi120.front.sepia.ceph.com
2024-04-12T22:43:11.871 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi156