2024-01-17T20:49:39.028 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-01-17T20:49:39.029 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-17T20:49:39.168 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f760dd8df70>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f760edf3c10>, signals=[15])
2024-01-17T20:49:39.169 INFO:teuthology.lock.ops:Start node 'smithi061.front.sepia.ceph.com' reimaging
2024-01-17T20:49:39.169 INFO:teuthology.lock.ops:Updating [smithi061.front.sepia.ceph.com]: reset os type and version on server
2024-01-17T20:49:39.170 INFO:teuthology.lock.ops:Updating smithi061.front.sepia.ceph.com on lock server
2024-01-17T20:49:39.193 INFO:teuthology.lock.ops:Start node 'smithi170.front.sepia.ceph.com' reimaging
2024-01-17T20:49:39.193 INFO:teuthology.lock.ops:Updating [smithi170.front.sepia.ceph.com]: reset os type and version on server
2024-01-17T20:49:39.193 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server
2024-01-17T20:49:39.320 DEBUG:teuthology.provision.fog.smithi061:Requesting image smithi_ubuntu_22.04 (ID 6)
2024-01-17T20:49:39.323 DEBUG:teuthology.provision.fog.smithi170:Requesting image smithi_ubuntu_22.04 (ID 6)
2024-01-17T20:49:39.363 INFO:teuthology.provision.fog.smithi061:Scheduling deploy of ubuntu 22.04
2024-01-17T20:49:39.368 INFO:teuthology.provision.fog.smithi170:Scheduling deploy of ubuntu 22.04
2024-01-17T20:49:39.583 INFO:teuthology.orchestra.console.smithi170:Power off
2024-01-17T20:49:39.584 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-01-17T20:49:39.597 INFO:teuthology.orchestra.console.smithi061:Power off
2024-01-17T20:49:39.597 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-01-17T20:49:39.605 DEBUG:teuthology.orchestra.console.smithi170:power off output: Chassis Power Control: Down/Off
2024-01-17T20:49:39.606 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:49:39.628 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is off
2024-01-17T20:49:39.729 INFO:teuthology.orchestra.console.smithi170:Power off completed
2024-01-17T20:49:39.829 INFO:teuthology.orchestra.console.smithi170:Power on
2024-01-17T20:49:39.829 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-01-17T20:49:39.851 DEBUG:teuthology.orchestra.console.smithi170:power on output: Chassis Power Control: Up/On
2024-01-17T20:49:39.851 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:49:39.878 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is off
2024-01-17T20:49:43.882 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:49:43.993 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on
2024-01-17T20:49:44.094 INFO:teuthology.orchestra.console.smithi170:Power on completed
2024-01-17T20:49:44.194 INFO:teuthology.provision.fog.smithi170:Waiting for deploy to finish
2024-01-17T20:49:59.662 DEBUG:teuthology.orchestra.console.smithi061:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2024-01-17T20:49:59.663 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-01-17T20:50:19.730 DEBUG:teuthology.orchestra.console.smithi061:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2024-01-17T20:50:19.730 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-01-17T20:50:39.802 DEBUG:teuthology.orchestra.console.smithi061:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2024-01-17T20:50:39.803 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:50:40.811 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:50:44.813 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:50:45.922 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:50:49.924 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:50:51.032 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:50:55.036 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:50:56.145 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:51:00.147 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:51:01.257 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:51:05.261 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:51:06.370 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:51:10.375 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:51:11.484 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:51:15.486 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:51:16.595 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:51:20.596 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:51:21.705 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:51:25.708 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:51:26.818 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:51:30.820 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:51:31.931 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:51:35.932 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:51:37.040 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:51:41.044 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:51:42.152 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:51:46.154 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:51:47.264 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:51:51.267 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:51:52.376 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:51:52.377 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2024-01-17T20:51:52.377 ERROR:teuthology.orchestra.console.smithi061:Failed to power off
2024-01-17T20:51:52.377 INFO:teuthology.orchestra.console.smithi061:Power on
2024-01-17T20:51:52.377 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-01-17T20:52:12.452 DEBUG:teuthology.orchestra.console.smithi061:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2024-01-17T20:52:12.453 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-01-17T20:52:32.529 DEBUG:teuthology.orchestra.console.smithi061:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2024-01-17T20:52:32.529 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-01-17T20:52:52.599 DEBUG:teuthology.orchestra.console.smithi061:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2024-01-17T20:52:52.599 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:52:53.608 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:52:57.611 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:52:58.720 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:53:00.025 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-17T20:53:02.724 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:53:03.832 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:53:07.837 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:53:08.945 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:53:12.950 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:53:14.059 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:53:18.063 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:53:18.524 WARNING:teuthology.provision.fog.smithi170:[Errno None] Unable to connect to port 22 on 172.21.15.170
2024-01-17T20:53:19.171 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:53:23.176 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:53:24.284 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:53:24.526 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-17T20:53:27.580 WARNING:teuthology.provision.fog.smithi170:[Errno None] Unable to connect to port 22 on 172.21.15.170
2024-01-17T20:53:28.286 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:53:29.394 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:53:33.396 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:53:33.582 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-17T20:53:34.505 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:53:36.636 WARNING:teuthology.provision.fog.smithi170:[Errno None] Unable to connect to port 22 on 172.21.15.170
2024-01-17T20:53:38.507 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:53:39.615 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:53:42.641 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-17T20:53:43.618 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:53:44.726 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:53:45.692 WARNING:teuthology.provision.fog.smithi170:[Errno None] Unable to connect to port 22 on 172.21.15.170
2024-01-17T20:53:48.728 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:53:49.836 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:53:51.695 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-17T20:53:53.839 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:53:54.748 WARNING:teuthology.provision.fog.smithi170:[Errno None] Unable to connect to port 22 on 172.21.15.170
2024-01-17T20:53:54.950 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:53:58.952 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:54:00.060 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:54:00.751 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-17T20:54:04.065 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:54:05.173 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:54:05.173 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2024-01-17T20:54:05.173 DEBUG:teuthology.provision.fog.smithi061:Canceling deploy task with ID 719625
2024-01-17T20:54:05.239 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi061
2024-01-17T20:54:05.240 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-17T20:54:05.561 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=8e3a091cc3a242168ba9e238288ddfd3
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi061
2024-01-17T20:54:05.564 INFO:teuthology.nuke:Checking targets against current locks
2024-01-17T20:54:05.632 DEBUG:teuthology.nuke:shortname: smithi061
2024-01-17T20:54:05.632 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-01-17T20:54:05.647 DEBUG:teuthology.nuke:shortname: smithi170
2024-01-17T20:54:05.647 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-01-17T20:54:05.650 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi061.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-01-17_20:20:37-fs-main-distro-default-smithi/7519273', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-17 20:49:37.989869', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG8NuJB1fW7JNXTUxAOA5cZYhMmvcAfm0JPn1vogOrF0hTNSvcXWbbDW+EIcD9tK8JQl4YgLN4huMMwZ60njGT0='}
2024-01-17T20:54:05.668 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-01-17_20:20:37-fs-main-distro-default-smithi/7519273', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-17 20:49:37.987935', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHMGKAAvm85zY0WTfd+r5hbeHhdrVRut6aNI7+LwGhdcWr02kG0aP1RbrqLwmMazDMmsUIk0G+LjPDDrvjfJVKw='}
2024-01-17T20:54:05.698 INFO:teuthology.orchestra.console.smithi061:Power off
2024-01-17T20:54:05.699 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-01-17T20:54:05.730 INFO:teuthology.orchestra.console.smithi170:Power off
2024-01-17T20:54:05.731 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-01-17T20:54:05.752 DEBUG:teuthology.orchestra.console.smithi170:power off output: Chassis Power Control: Down/Off
2024-01-17T20:54:05.752 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:54:05.773 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on
2024-01-17T20:54:09.778 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:54:09.887 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is off
2024-01-17T20:54:09.987 INFO:teuthology.orchestra.console.smithi170:Power off completed
2024-01-17T20:54:10.147 INFO:teuthology.lock.ops:unlocked: smithi170.front.sepia.ceph.com
2024-01-17T20:54:25.772 DEBUG:teuthology.orchestra.console.smithi061:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2024-01-17T20:54:25.773 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-01-17T20:54:45.846 DEBUG:teuthology.orchestra.console.smithi061:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2024-01-17T20:54:45.846 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-01-17T20:55:00.769 WARNING:teuthology.provision.fog.smithi170:timed out
2024-01-17T20:55:05.915 DEBUG:teuthology.orchestra.console.smithi061:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2024-01-17T20:55:05.915 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:55:06.771 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-17T20:55:06.924 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:55:10.925 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:55:12.034 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:55:16.036 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:55:17.146 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:55:21.150 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:55:22.260 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:55:25.244 WARNING:teuthology.provision.fog.smithi170:[Errno None] Unable to connect to port 22 on 172.21.15.170
2024-01-17T20:55:26.262 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:55:27.370 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:55:31.250 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-17T20:55:31.372 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:55:32.482 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:55:34.300 WARNING:teuthology.provision.fog.smithi170:[Errno None] Unable to connect to port 22 on 172.21.15.170
2024-01-17T20:55:36.484 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:55:37.593 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:55:40.305 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-17T20:55:41.595 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:55:42.704 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:55:46.708 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:55:47.817 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:55:50.588 WARNING:teuthology.provision.fog.smithi170:[Errno None] Unable to connect to port 22 on 172.21.15.170
2024-01-17T20:55:51.820 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:55:52.929 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:55:56.591 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-17T20:55:56.931 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:55:58.040 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:55:59.644 WARNING:teuthology.provision.fog.smithi170:[Errno None] Unable to connect to port 22 on 172.21.15.170
2024-01-17T20:56:02.044 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:56:03.153 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:56:05.646 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-17T20:56:06.344 WARNING:teuthology.provision.fog.smithi170:[Errno None] Unable to connect to port 22 on 172.21.15.170
2024-01-17T20:56:07.156 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:56:08.266 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:56:12.270 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:56:12.346 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-17T20:56:12.482 ERROR:teuthology.orchestra.connection:Error authenticating with smithi170.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-01-17T20:56:13.379 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:56:16.620 ERROR:teuthology.orchestra.connection:Error authenticating with smithi170.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-01-17T20:56:17.380 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:56:18.490 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:56:18.491 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2024-01-17T20:56:18.491 ERROR:teuthology.orchestra.console.smithi061:Failed to power off
2024-01-17T20:56:18.651 INFO:teuthology.lock.ops:unlocked: smithi061.front.sepia.ceph.com
2024-01-17T20:56:18.791 ERROR:teuthology.dispatcher.supervisor:Reimage failed 10 times ... marking machine down
2024-01-17T20:56:18.920 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi061