2023-11-04T04:24:53.558 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2
2023-11-04T04:24:53.560 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T04:24:53.701 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7faf7cf3cb50>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7faf7ccd9b80>, signals=[15])
2023-11-04T04:24:53.702 INFO:teuthology.lock.ops:Start node 'smithi076.front.sepia.ceph.com' reimaging
2023-11-04T04:24:53.702 INFO:teuthology.lock.ops:Updating [smithi076.front.sepia.ceph.com]: reset os type and version on server
2023-11-04T04:24:53.703 INFO:teuthology.lock.ops:Updating smithi076.front.sepia.ceph.com on lock server
2023-11-04T04:24:53.723 INFO:teuthology.lock.ops:Start node 'smithi123.front.sepia.ceph.com' reimaging
2023-11-04T04:24:53.723 INFO:teuthology.lock.ops:Updating [smithi123.front.sepia.ceph.com]: reset os type and version on server
2023-11-04T04:24:53.724 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server
2023-11-04T04:24:53.725 DEBUG:teuthology.provision.fog.smithi076:FOG request: GET /host {"name": "smithi076"}
2023-11-04T04:24:53.747 DEBUG:teuthology.provision.fog.smithi123:FOG request: GET /host {"name": "smithi123"}
2023-11-04T04:24:53.809 DEBUG:teuthology.provision.fog.smithi076:FOG request: GET /image {"name": "smithi_ubuntu_22.04"}
2023-11-04T04:24:53.826 DEBUG:teuthology.provision.fog.smithi123:FOG request: GET /image {"name": "smithi_ubuntu_22.04"}
2023-11-04T04:24:53.846 DEBUG:teuthology.provision.fog.smithi076:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-11-04T04:24:53.846 DEBUG:teuthology.provision.fog.smithi076:FOG request: PUT /host/181 {"imageID": 6}
2023-11-04T04:24:53.870 DEBUG:teuthology.provision.fog.smithi123:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-11-04T04:24:53.870 DEBUG:teuthology.provision.fog.smithi123:FOG request: PUT /host/228 {"imageID": 6}
2023-11-04T04:24:53.895 INFO:teuthology.provision.fog.smithi076:Scheduling deploy of ubuntu 22.04
2023-11-04T04:24:53.895 DEBUG:teuthology.provision.fog.smithi076:FOG request: GET /task/active None
2023-11-04T04:24:53.916 INFO:teuthology.provision.fog.smithi123:Scheduling deploy of ubuntu 22.04
2023-11-04T04:24:53.916 DEBUG:teuthology.provision.fog.smithi123:FOG request: GET /task/active None
2023-11-04T04:24:53.952 DEBUG:teuthology.provision.fog.smithi076:FOG request: GET /tasktype {"name": "deploy"}
2023-11-04T04:24:53.973 DEBUG:teuthology.provision.fog.smithi123:FOG request: GET /tasktype {"name": "deploy"}
2023-11-04T04:24:53.992 DEBUG:teuthology.provision.fog.smithi076:FOG request: POST /host/181/task {"taskTypeID": 1}
2023-11-04T04:24:54.009 DEBUG:teuthology.provision.fog.smithi123:FOG request: POST /host/228/task {"taskTypeID": 1}
2023-11-04T04:24:54.047 DEBUG:teuthology.provision.fog.smithi076:FOG request: GET /task/active None
2023-11-04T04:24:54.064 DEBUG:teuthology.provision.fog.smithi123:FOG request: GET /task/active None
2023-11-04T04:24:54.140 INFO:teuthology.orchestra.console.smithi076:Power off
2023-11-04T04:24:54.141 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:24:54.155 INFO:teuthology.orchestra.console.smithi123:Power off
2023-11-04T04:24:54.155 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:24:54.197 DEBUG:teuthology.orchestra.console.smithi076:power off output: Chassis Power Control: Down/Off
2023-11-04T04:24:54.197 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:24:54.219 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is off
2023-11-04T04:24:54.320 INFO:teuthology.orchestra.console.smithi076:Power off completed
2023-11-04T04:24:54.421 INFO:teuthology.orchestra.console.smithi076:Power on
2023-11-04T04:24:54.422 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T04:24:54.442 DEBUG:teuthology.orchestra.console.smithi076:power on output: Chassis Power Control: Up/On
2023-11-04T04:24:54.443 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:24:54.465 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is off
2023-11-04T04:24:58.466 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:24:58.575 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is off
2023-11-04T04:25:02.578 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:25:02.687 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on
2023-11-04T04:25:02.788 INFO:teuthology.orchestra.console.smithi076:Power on completed
2023-11-04T04:25:02.889 INFO:teuthology.provision.fog.smithi076:Waiting for deploy to finish
2023-11-04T04:25:02.890 DEBUG:teuthology.provision.fog.smithi076:FOG request: GET /task/active None
2023-11-04T04:25:14.225 DEBUG:teuthology.orchestra.console.smithi123:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:25:14.225 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:25:17.966 DEBUG:teuthology.provision.fog.smithi076:FOG request: GET /task/active None
2023-11-04T04:25:33.058 DEBUG:teuthology.provision.fog.smithi076:FOG request: GET /task/active None
2023-11-04T04:25:34.296 DEBUG:teuthology.orchestra.console.smithi123:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:25:34.296 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:25:48.142 DEBUG:teuthology.provision.fog.smithi076:FOG request: GET /task/active None
2023-11-04T04:25:54.368 DEBUG:teuthology.orchestra.console.smithi123:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:25:54.369 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:25:55.376 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:25:59.381 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:26:00.490 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:26:03.238 DEBUG:teuthology.provision.fog.smithi076:FOG request: GET /task/active None
2023-11-04T04:26:04.493 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:26:05.602 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:26:09.606 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:26:10.716 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:26:14.718 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:26:15.826 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:26:18.323 DEBUG:teuthology.provision.fog.smithi076:FOG request: GET /task/active None
2023-11-04T04:26:19.829 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:26:20.938 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:26:24.943 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:26:26.052 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:26:30.054 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:26:31.165 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:26:33.415 DEBUG:teuthology.provision.fog.smithi076:FOG request: GET /task/active None
2023-11-04T04:26:35.168 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:26:36.277 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:26:40.282 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:26:41.391 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:26:45.394 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:26:46.503 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:26:48.512 DEBUG:teuthology.provision.fog.smithi076:FOG request: GET /task/active None
2023-11-04T04:26:50.506 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:26:51.616 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:26:55.618 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:26:56.730 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:27:00.734 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:27:01.842 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:27:03.594 DEBUG:teuthology.provision.fog.smithi076:FOG request: GET /task/active None
2023-11-04T04:27:05.846 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:27:06.956 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:27:06.957 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T04:27:06.957 ERROR:teuthology.orchestra.console.smithi123:Failed to power off
2023-11-04T04:27:06.957 INFO:teuthology.orchestra.console.smithi123:Power on
2023-11-04T04:27:06.958 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T04:27:18.706 DEBUG:teuthology.provision.fog.smithi076:FOG request: GET /task/active None
2023-11-04T04:27:27.024 DEBUG:teuthology.orchestra.console.smithi123:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:27:27.025 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T04:27:33.810 DEBUG:teuthology.provision.fog.smithi076:FOG request: GET /task/active None
2023-11-04T04:27:47.097 DEBUG:teuthology.orchestra.console.smithi123:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:27:47.098 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T04:27:48.910 DEBUG:teuthology.provision.fog.smithi076:FOG request: GET /task/active None
2023-11-04T04:28:04.034 DEBUG:teuthology.provision.fog.smithi076:FOG request: GET /task/active None
2023-11-04T04:28:07.168 DEBUG:teuthology.orchestra.console.smithi123:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:28:07.168 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:28:08.177 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:28:12.182 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:28:13.291 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:28:17.294 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:28:18.405 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:28:19.161 DEBUG:teuthology.provision.fog.smithi076:FOG request: GET /task/active None
2023-11-04T04:28:22.410 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:28:23.518 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:28:27.522 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:28:28.631 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:28:32.634 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:28:33.742 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:28:34.295 DEBUG:teuthology.provision.fog.smithi076:FOG request: GET /task/active None
2023-11-04T04:28:34.407 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:28:37.746 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:28:38.855 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:28:42.858 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:28:43.967 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:28:47.970 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:28:49.080 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:28:52.731 WARNING:teuthology.provision.fog.smithi076:[Errno None] Unable to connect to port 22 on 172.21.15.76
2023-11-04T04:28:53.081 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:28:54.190 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:28:58.194 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:28:58.734 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:28:59.304 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:29:01.790 WARNING:teuthology.provision.fog.smithi076:[Errno None] Unable to connect to port 22 on 172.21.15.76
2023-11-04T04:29:03.306 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:29:04.415 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:29:07.795 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:29:08.417 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:29:09.526 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:29:10.878 WARNING:teuthology.provision.fog.smithi076:[Errno None] Unable to connect to port 22 on 172.21.15.76
2023-11-04T04:29:13.530 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:29:14.638 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:29:16.882 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:29:18.642 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:29:19.750 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:29:19.751 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T04:29:19.751 DEBUG:teuthology.provision.fog.smithi123:Canceling deploy task with ID 582469
2023-11-04T04:29:19.751 DEBUG:teuthology.provision.fog.smithi123:FOG request: DELETE /task/cancel {"id": 582469}
2023-11-04T04:29:19.804 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi123
2023-11-04T04:29:19.804 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T04:29:19.951 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=877cc2b5436d4c2ba891432be4c3c802
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi123
2023-11-04T04:29:19.954 INFO:teuthology.nuke:Checking targets against current locks
2023-11-04T04:29:19.977 WARNING:teuthology.provision.fog.smithi076:[Errno None] Unable to connect to port 22 on 172.21.15.76
2023-11-04T04:29:20.028 DEBUG:teuthology.nuke:shortname: smithi076
2023-11-04T04:29:20.029 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-04T04:29:20.050 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi076.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/ivancich-2023-11-03_20:37:04-rgw-wip-eric-testing-1-distro-default-smithi/7446014', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-04 04:24:52.744489', 'locked_by': 'scheduled_ivancich@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL/XOD0qettnAueqTNK3ImRArACupJr7reUpKoylQu8ZPHSrv/1MDcJB0soPim4XTgtqVcXAXwgRe/haESHcI3g='}
2023-11-04T04:29:20.054 DEBUG:teuthology.nuke:shortname: smithi123
2023-11-04T04:29:20.054 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-04T04:29:20.076 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/ivancich-2023-11-03_20:37:04-rgw-wip-eric-testing-1-distro-default-smithi/7446014', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-04 04:24:52.746445', 'locked_by': 'scheduled_ivancich@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEQnccEOO81GbxL1Hh9Rh+5xrMeM2TkjYhFwig3fteY1Kijuki/tOyfznpleG1gHUfe6UZTziY6YZWuiR9SXit0='}
2023-11-04T04:29:20.112 INFO:teuthology.orchestra.console.smithi076:Power off
2023-11-04T04:29:20.113 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:29:20.127 INFO:teuthology.orchestra.console.smithi123:Power off
2023-11-04T04:29:20.128 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:29:20.138 DEBUG:teuthology.orchestra.console.smithi076:power off output: Chassis Power Control: Down/Off
2023-11-04T04:29:20.139 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:29:20.160 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on
2023-11-04T04:29:24.166 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:29:24.275 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is off
2023-11-04T04:29:24.376 INFO:teuthology.orchestra.console.smithi076:Power off completed
2023-11-04T04:29:24.529 INFO:teuthology.lock.ops:unlocked: smithi076.front.sepia.ceph.com
2023-11-04T04:29:25.981 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:29:29.050 WARNING:teuthology.provision.fog.smithi076:[Errno None] Unable to connect to port 22 on 172.21.15.76
2023-11-04T04:29:35.054 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:29:38.110 WARNING:teuthology.provision.fog.smithi076:[Errno None] Unable to connect to port 22 on 172.21.15.76
2023-11-04T04:29:40.196 DEBUG:teuthology.orchestra.console.smithi123:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:29:40.196 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:29:44.115 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:29:47.194 WARNING:teuthology.provision.fog.smithi076:[Errno None] Unable to connect to port 22 on 172.21.15.76
2023-11-04T04:29:53.199 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:29:56.250 WARNING:teuthology.provision.fog.smithi076:[Errno None] Unable to connect to port 22 on 172.21.15.76
2023-11-04T04:30:00.268 DEBUG:teuthology.orchestra.console.smithi123:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:30:00.268 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:30:02.253 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:30:05.306 WARNING:teuthology.provision.fog.smithi076:[Errno None] Unable to connect to port 22 on 172.21.15.76
2023-11-04T04:30:11.313 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:30:14.398 WARNING:teuthology.provision.fog.smithi076:[Errno None] Unable to connect to port 22 on 172.21.15.76
2023-11-04T04:30:20.337 DEBUG:teuthology.orchestra.console.smithi123:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:30:20.338 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:30:20.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:30:21.347 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:30:23.450 WARNING:teuthology.provision.fog.smithi076:[Errno None] Unable to connect to port 22 on 172.21.15.76
2023-11-04T04:30:25.349 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:30:26.462 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:30:29.458 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:30:30.464 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:30:31.574 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:30:32.542 WARNING:teuthology.provision.fog.smithi076:[Errno None] Unable to connect to port 22 on 172.21.15.76
2023-11-04T04:30:35.578 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:30:36.687 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:30:38.546 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:30:40.690 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:30:41.630 WARNING:teuthology.provision.fog.smithi076:[Errno None] Unable to connect to port 22 on 172.21.15.76
2023-11-04T04:30:41.799 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:30:45.803 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:30:46.912 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:30:47.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:30:50.714 WARNING:teuthology.provision.fog.smithi076:[Errno None] Unable to connect to port 22 on 172.21.15.76
2023-11-04T04:30:50.914 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:30:52.022 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:30:56.026 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:30:56.717 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:30:57.136 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:30:59.770 WARNING:teuthology.provision.fog.smithi076:[Errno None] Unable to connect to port 22 on 172.21.15.76
2023-11-04T04:31:01.138 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:31:02.247 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:31:05.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:31:06.249 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:31:07.358 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:31:11.363 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:31:12.473 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:31:16.090 WARNING:teuthology.provision.fog.smithi076:[Errno None] Unable to connect to port 22 on 172.21.15.76
2023-11-04T04:31:16.475 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:31:17.585 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:31:21.587 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:31:22.093 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:31:22.697 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:31:25.146 WARNING:teuthology.provision.fog.smithi076:[Errno None] Unable to connect to port 22 on 172.21.15.76
2023-11-04T04:31:26.700 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:31:27.809 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:31:31.151 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:31:31.280 ERROR:teuthology.orchestra.connection:Error authenticating with smithi076.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2023-11-04T04:31:31.811 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:31:32.920 DEBUG:teuthology.orchestra.console.smithi123:check power output: 
2023-11-04T04:31:32.921 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T04:31:32.921 ERROR:teuthology.orchestra.console.smithi123:Failed to power off
2023-11-04T04:31:33.081 INFO:teuthology.lock.ops:unlocked: smithi123.front.sepia.ceph.com
2023-11-04T04:31:34.738 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi123