2024-03-11T23:50:02.583 INFO:root:teuthology version: 0.0.1.dev262+ge691533
2024-03-11T23:50:02.584 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-11T23:50:02.784 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f602edc0f10>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f602fe20c10>, signals=[15])
2024-03-11T23:50:02.785 INFO:teuthology.lock.ops:Start node 'smithi027.front.sepia.ceph.com' reimaging
2024-03-11T23:50:02.785 INFO:teuthology.lock.ops:Updating [smithi027.front.sepia.ceph.com]: reset os type and version on server
2024-03-11T23:50:02.785 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server
2024-03-11T23:50:02.811 INFO:teuthology.lock.ops:Start node 'smithi038.front.sepia.ceph.com' reimaging
2024-03-11T23:50:02.812 INFO:teuthology.lock.ops:Updating [smithi038.front.sepia.ceph.com]: reset os type and version on server
2024-03-11T23:50:02.812 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server
2024-03-11T23:50:02.827 INFO:teuthology.lock.ops:Start node 'smithi045.front.sepia.ceph.com' reimaging
2024-03-11T23:50:02.828 INFO:teuthology.lock.ops:Updating [smithi045.front.sepia.ceph.com]: reset os type and version on server
2024-03-11T23:50:02.828 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server
2024-03-11T23:50:02.844 INFO:teuthology.lock.ops:Start node 'smithi071.front.sepia.ceph.com' reimaging
2024-03-11T23:50:02.844 INFO:teuthology.lock.ops:Updating [smithi071.front.sepia.ceph.com]: reset os type and version on server
2024-03-11T23:50:02.844 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server
2024-03-11T23:50:02.927 DEBUG:teuthology.provision.fog.smithi027:Requesting image smithi_centos_9.stream (ID 10)
2024-03-11T23:50:02.954 DEBUG:teuthology.provision.fog.smithi038:Requesting image smithi_centos_9.stream (ID 10)
2024-03-11T23:50:02.971 DEBUG:teuthology.provision.fog.smithi045:Requesting image smithi_centos_9.stream (ID 10)
2024-03-11T23:50:02.974 DEBUG:teuthology.provision.fog.smithi071:Requesting image smithi_centos_9.stream (ID 10)
2024-03-11T23:50:02.980 INFO:teuthology.provision.fog.smithi027:Scheduling deploy of centos 9.stream
2024-03-11T23:50:03.000 INFO:teuthology.provision.fog.smithi038:Scheduling deploy of centos 9.stream
2024-03-11T23:50:03.023 INFO:teuthology.provision.fog.smithi045:Scheduling deploy of centos 9.stream
2024-03-11T23:50:03.026 INFO:teuthology.provision.fog.smithi071:Scheduling deploy of centos 9.stream
2024-03-11T23:50:03.288 INFO:teuthology.orchestra.console.smithi027:Power off
2024-03-11T23:50:03.289 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-11T23:50:03.305 INFO:teuthology.orchestra.console.smithi038:Power off
2024-03-11T23:50:03.305 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-11T23:50:03.316 DEBUG:teuthology.orchestra.console.smithi027:power off output: Chassis Power Control: Down/Off
2024-03-11T23:50:03.316 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:50:03.329 INFO:teuthology.orchestra.console.smithi045:Power off
2024-03-11T23:50:03.329 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-11T23:50:03.343 INFO:teuthology.orchestra.console.smithi071:Power off
2024-03-11T23:50:03.343 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-11T23:50:03.349 DEBUG:teuthology.orchestra.console.smithi038:power off output: Chassis Power Control: Down/Off
2024-03-11T23:50:03.350 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:50:03.357 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is off
2024-03-11T23:50:03.359 DEBUG:teuthology.orchestra.console.smithi045:power off output: Chassis Power Control: Down/Off
2024-03-11T23:50:03.359 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:50:03.367 DEBUG:teuthology.orchestra.console.smithi071:power off output: Chassis Power Control: Down/Off
2024-03-11T23:50:03.368 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:50:03.375 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is off
2024-03-11T23:50:03.381 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on
2024-03-11T23:50:03.387 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is off
2024-03-11T23:50:03.458 INFO:teuthology.orchestra.console.smithi027:Power off completed
2024-03-11T23:50:03.476 INFO:teuthology.orchestra.console.smithi038:Power off completed
2024-03-11T23:50:03.488 INFO:teuthology.orchestra.console.smithi071:Power off completed
2024-03-11T23:50:03.559 INFO:teuthology.orchestra.console.smithi027:Power on
2024-03-11T23:50:03.560 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-03-11T23:50:03.577 INFO:teuthology.orchestra.console.smithi038:Power on
2024-03-11T23:50:03.577 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-03-11T23:50:03.585 DEBUG:teuthology.orchestra.console.smithi027:power on output: Chassis Power Control: Up/On
2024-03-11T23:50:03.586 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:50:03.593 INFO:teuthology.orchestra.console.smithi071:Power on
2024-03-11T23:50:03.593 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-03-11T23:50:03.601 DEBUG:teuthology.orchestra.console.smithi038:power on output: Chassis Power Control: Up/On
2024-03-11T23:50:03.601 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:50:03.609 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is off
2024-03-11T23:50:03.613 DEBUG:teuthology.orchestra.console.smithi071:power on output: Chassis Power Control: Up/On
2024-03-11T23:50:03.613 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:50:03.623 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is off
2024-03-11T23:50:03.632 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is off
2024-03-11T23:50:07.385 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:50:07.493 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off
2024-03-11T23:50:07.594 INFO:teuthology.orchestra.console.smithi045:Power off completed
2024-03-11T23:50:07.610 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:50:07.624 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:50:07.633 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:50:07.695 INFO:teuthology.orchestra.console.smithi045:Power on
2024-03-11T23:50:07.695 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-03-11T23:50:07.717 DEBUG:teuthology.orchestra.console.smithi045:power on output: Chassis Power Control: Up/On
2024-03-11T23:50:07.717 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:50:07.725 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is off
2024-03-11T23:50:07.732 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on
2024-03-11T23:50:07.737 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off
2024-03-11T23:50:07.741 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is off
2024-03-11T23:50:07.834 INFO:teuthology.orchestra.console.smithi038:Power on completed
2024-03-11T23:50:07.934 INFO:teuthology.provision.fog.smithi038:Waiting for deploy to finish
2024-03-11T23:50:11.726 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:50:11.737 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:50:11.745 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:50:11.836 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on
2024-03-11T23:50:11.847 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off
2024-03-11T23:50:11.852 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on
2024-03-11T23:50:11.937 INFO:teuthology.orchestra.console.smithi027:Power on completed
2024-03-11T23:50:11.953 INFO:teuthology.orchestra.console.smithi071:Power on completed
2024-03-11T23:50:12.038 INFO:teuthology.provision.fog.smithi027:Waiting for deploy to finish
2024-03-11T23:50:12.054 INFO:teuthology.provision.fog.smithi071:Waiting for deploy to finish
2024-03-11T23:50:15.850 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:50:15.959 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off
2024-03-11T23:50:19.961 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:50:20.070 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off
2024-03-11T23:50:24.072 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:50:24.181 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off
2024-03-11T23:50:28.183 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:50:28.292 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off
2024-03-11T23:50:32.296 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:50:32.405 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off
2024-03-11T23:50:36.409 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:50:36.519 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off
2024-03-11T23:50:40.522 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:50:40.630 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off
2024-03-11T23:50:44.633 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:50:44.742 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off
2024-03-11T23:50:48.745 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:50:48.854 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off
2024-03-11T23:50:52.858 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:50:52.967 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off
2024-03-11T23:50:56.971 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:50:57.080 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off
2024-03-11T23:51:01.081 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:51:01.189 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off
2024-03-11T23:51:05.193 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:51:05.302 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off
2024-03-11T23:51:05.302 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2024-03-11T23:51:05.402 DEBUG:teuthology.provision.fog.smithi045:Canceling deploy task with ID 850846
2024-03-11T23:51:05.509 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi045
2024-03-11T23:51:05.510 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-11T23:51:05.798 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=2a6950e30f624131b1272ce073e637da
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi045
2024-03-11T23:51:05.800 INFO:teuthology.nuke:Checking targets against current locks
2024-03-11T23:51:05.865 DEBUG:teuthology.nuke:shortname: smithi027
2024-03-11T23:51:05.865 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-03-11T23:51:05.887 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:32:02-powercycle-squid-distro-default-smithi/7592104', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 23:50:01.481767', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAMzloCQ++/Ekt/Ztuby7XylhTr1liEeYDTICceEFC8HVcjz1cnz51NC6Zyk3C5Q13BpTcNx6/awlQiGwyUFWcs='}
2024-03-11T23:51:05.888 DEBUG:teuthology.nuke:shortname: smithi038
2024-03-11T23:51:05.888 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-03-11T23:51:05.908 DEBUG:teuthology.nuke:shortname: smithi045
2024-03-11T23:51:05.908 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-03-11T23:51:05.914 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:32:02-powercycle-squid-distro-default-smithi/7592104', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 23:50:01.482928', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBObxlPBPvI/VXHWtagxrGMF4gTyODgggb1hskuZVy9qc6rdcxIaa+jBbjNR+Gv5lNYZwVpPjF7xKGDGoQglre98='}
2024-03-11T23:51:05.918 DEBUG:teuthology.nuke:shortname: smithi071
2024-03-11T23:51:05.918 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-03-11T23:51:05.927 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:32:02-powercycle-squid-distro-default-smithi/7592104', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 23:50:01.484055', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ3OFEZ2sVMqH8C/7k9tf2J34hK/qxpUvOxDj4tajeboS3BH8rMK4nLNwyiHQ8D7Y3aUTFygMn4rB3Z38GPFeP4='}
2024-03-11T23:51:05.941 INFO:teuthology.orchestra.console.smithi027:Power off
2024-03-11T23:51:05.941 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-11T23:51:05.952 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:32:02-powercycle-squid-distro-default-smithi/7592104', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 23:50:01.480223', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIc98BRk/ghN7oUQymf8pnmSeGmJdZosZSE3T2uwEadnewLSYQRNJw5PAtz9p+Rk5Y4zkBJyaezi6PHvJn6ohCQ='}
2024-03-11T23:51:05.964 DEBUG:teuthology.orchestra.console.smithi027:power off output: Chassis Power Control: Down/Off
2024-03-11T23:51:05.964 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:51:05.976 INFO:teuthology.orchestra.console.smithi038:Power off
2024-03-11T23:51:05.976 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-11T23:51:05.990 INFO:teuthology.orchestra.console.smithi045:Power off
2024-03-11T23:51:05.990 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-11T23:51:06.002 DEBUG:teuthology.orchestra.console.smithi038:power off output: Chassis Power Control: Down/Off
2024-03-11T23:51:06.002 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:51:06.010 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on
2024-03-11T23:51:06.012 DEBUG:teuthology.orchestra.console.smithi045:power off output: Chassis Power Control: Down/Off
2024-03-11T23:51:06.012 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:51:06.029 INFO:teuthology.orchestra.console.smithi071:Power off
2024-03-11T23:51:06.029 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-11T23:51:06.037 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off
2024-03-11T23:51:06.037 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on
2024-03-11T23:51:06.049 DEBUG:teuthology.orchestra.console.smithi071:power off output: Chassis Power Control: Down/Off
2024-03-11T23:51:06.049 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:51:06.071 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on
2024-03-11T23:51:06.138 INFO:teuthology.orchestra.console.smithi045:Power off completed
2024-03-11T23:51:06.306 INFO:teuthology.lock.ops:unlocked: smithi045.front.sepia.ceph.com
2024-03-11T23:51:10.014 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:51:10.038 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:51:10.071 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-11T23:51:10.124 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is off
2024-03-11T23:51:10.147 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is off
2024-03-11T23:51:10.183 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is off
2024-03-11T23:51:10.225 INFO:teuthology.orchestra.console.smithi027:Power off completed
2024-03-11T23:51:10.248 INFO:teuthology.orchestra.console.smithi038:Power off completed
2024-03-11T23:51:10.283 INFO:teuthology.orchestra.console.smithi071:Power off completed
2024-03-11T23:51:10.380 INFO:teuthology.lock.ops:unlocked: smithi027.front.sepia.ceph.com
2024-03-11T23:51:10.396 INFO:teuthology.lock.ops:unlocked: smithi038.front.sepia.ceph.com
2024-03-11T23:51:10.430 INFO:teuthology.lock.ops:unlocked: smithi071.front.sepia.ceph.com
2024-03-11T23:51:10.935 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi045