2023-11-04T04:07:41.161 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2
2023-11-04T04:07:41.162 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T04:07:41.306 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7fc42f8dbb80>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7fc4303cd670>, signals=[15])
2023-11-04T04:07:41.308 INFO:teuthology.lock.ops:Start node 'smithi097.front.sepia.ceph.com' reimaging
2023-11-04T04:07:41.309 INFO:teuthology.lock.ops:Updating [smithi097.front.sepia.ceph.com]: reset os type and version on server
2023-11-04T04:07:41.309 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server
2023-11-04T04:07:41.333 INFO:teuthology.lock.ops:Start node 'smithi132.front.sepia.ceph.com' reimaging
2023-11-04T04:07:41.334 INFO:teuthology.lock.ops:Updating [smithi132.front.sepia.ceph.com]: reset os type and version on server
2023-11-04T04:07:41.334 INFO:teuthology.lock.ops:Updating smithi132.front.sepia.ceph.com on lock server
2023-11-04T04:07:41.335 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /host {"name": "smithi097"}
2023-11-04T04:07:41.358 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /host {"name": "smithi132"}
2023-11-04T04:07:41.404 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /image {"name": "smithi_ubuntu_22.04"}
2023-11-04T04:07:41.428 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /image {"name": "smithi_ubuntu_22.04"}
2023-11-04T04:07:41.446 DEBUG:teuthology.provision.fog.smithi097:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-11-04T04:07:41.446 DEBUG:teuthology.provision.fog.smithi097:FOG request: PUT /host/202 {"imageID": 6}
2023-11-04T04:07:41.469 DEBUG:teuthology.provision.fog.smithi132:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-11-04T04:07:41.469 DEBUG:teuthology.provision.fog.smithi132:FOG request: PUT /host/237 {"imageID": 6}
2023-11-04T04:07:41.497 INFO:teuthology.provision.fog.smithi097:Scheduling deploy of ubuntu 22.04
2023-11-04T04:07:41.497 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None
2023-11-04T04:07:41.527 INFO:teuthology.provision.fog.smithi132:Scheduling deploy of ubuntu 22.04
2023-11-04T04:07:41.528 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None
2023-11-04T04:07:41.597 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /tasktype {"name": "deploy"}
2023-11-04T04:07:41.602 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /tasktype {"name": "deploy"}
2023-11-04T04:07:41.637 DEBUG:teuthology.provision.fog.smithi132:FOG request: POST /host/237/task {"taskTypeID": 1}
2023-11-04T04:07:41.649 DEBUG:teuthology.provision.fog.smithi097:FOG request: POST /host/202/task {"taskTypeID": 1}
2023-11-04T04:07:41.697 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None
2023-11-04T04:07:41.701 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None
2023-11-04T04:07:41.823 INFO:teuthology.orchestra.console.smithi132:Power off
2023-11-04T04:07:41.824 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:07:41.838 INFO:teuthology.orchestra.console.smithi097:Power off
2023-11-04T04:07:41.838 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:07:41.846 DEBUG:teuthology.orchestra.console.smithi132:power off output: Chassis Power Control: Down/Off
2023-11-04T04:07:41.846 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:07:41.867 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is off
2023-11-04T04:07:41.968 INFO:teuthology.orchestra.console.smithi132:Power off completed
2023-11-04T04:07:42.069 INFO:teuthology.orchestra.console.smithi132:Power on
2023-11-04T04:07:42.069 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T04:07:42.090 DEBUG:teuthology.orchestra.console.smithi132:power on output: Chassis Power Control: Up/On
2023-11-04T04:07:42.090 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:07:42.110 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is off
2023-11-04T04:07:46.114 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:07:46.222 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is on
2023-11-04T04:07:46.323 INFO:teuthology.orchestra.console.smithi132:Power on completed
2023-11-04T04:07:46.424 INFO:teuthology.provision.fog.smithi132:Waiting for deploy to finish
2023-11-04T04:07:46.425 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None
2023-11-04T04:08:01.532 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None
2023-11-04T04:08:01.906 DEBUG:teuthology.orchestra.console.smithi097:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:08:01.906 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:08:16.647 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None
2023-11-04T04:08:21.980 DEBUG:teuthology.orchestra.console.smithi097:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:08:21.981 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:08:31.758 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None
2023-11-04T04:08:42.054 DEBUG:teuthology.orchestra.console.smithi097:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:08:42.054 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:08:43.062 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:08:46.838 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None
2023-11-04T04:08:47.063 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:08:48.172 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:08:52.176 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:08:53.285 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:08:57.289 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:08:58.398 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:09:01.933 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None
2023-11-04T04:09:02.400 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:09:03.509 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:09:07.514 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:09:08.623 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:09:12.626 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:09:13.734 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:09:17.021 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None
2023-11-04T04:09:17.737 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:09:18.845 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:09:22.850 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:09:23.960 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:09:27.965 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:09:29.074 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:09:32.107 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None
2023-11-04T04:09:33.076 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:09:34.185 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:09:38.187 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:09:39.296 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:09:43.298 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:09:44.407 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:09:47.223 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None
2023-11-04T04:09:48.409 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:09:49.519 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:09:53.522 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:09:54.631 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:09:54.632 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T04:09:54.633 ERROR:teuthology.orchestra.console.smithi097:Failed to power off
2023-11-04T04:09:54.633 INFO:teuthology.orchestra.console.smithi097:Power on
2023-11-04T04:09:54.633 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T04:10:02.330 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None
2023-11-04T04:10:14.704 DEBUG:teuthology.orchestra.console.smithi097:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:10:14.705 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T04:10:17.429 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None
2023-11-04T04:10:32.539 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None
2023-11-04T04:10:34.772 DEBUG:teuthology.orchestra.console.smithi097:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:10:34.772 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-04T04:10:47.642 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None
2023-11-04T04:10:54.841 DEBUG:teuthology.orchestra.console.smithi097:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:10:54.842 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:10:55.850 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:10:59.854 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:11:00.963 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:11:02.754 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None
2023-11-04T04:11:02.842 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:11:04.966 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:11:05.914 WARNING:teuthology.provision.fog.smithi132:[Errno None] Unable to connect to port 22 on 172.21.15.132
2023-11-04T04:11:06.075 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:11:10.077 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:11:11.186 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:11:11.917 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:11:14.970 WARNING:teuthology.provision.fog.smithi132:[Errno None] Unable to connect to port 22 on 172.21.15.132
2023-11-04T04:11:15.188 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:11:16.297 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:11:20.302 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:11:20.973 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:11:21.411 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:11:24.030 WARNING:teuthology.provision.fog.smithi132:[Errno None] Unable to connect to port 22 on 172.21.15.132
2023-11-04T04:11:25.414 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:11:26.522 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:11:30.035 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:11:30.524 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:11:31.633 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:11:33.114 WARNING:teuthology.provision.fog.smithi132:[Errno None] Unable to connect to port 22 on 172.21.15.132
2023-11-04T04:11:35.635 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:11:36.748 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:11:39.119 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:11:40.750 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:11:41.859 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:11:42.170 WARNING:teuthology.provision.fog.smithi132:[Errno None] Unable to connect to port 22 on 172.21.15.132
2023-11-04T04:11:45.862 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:11:46.974 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:11:48.173 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:11:50.978 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:11:52.090 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:11:56.094 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:11:57.204 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:12:01.206 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:12:02.318 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:12:06.322 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:12:07.434 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:12:07.434 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T04:12:07.435 DEBUG:teuthology.provision.fog.smithi097:Canceling deploy task with ID 582385
2023-11-04T04:12:07.435 DEBUG:teuthology.provision.fog.smithi097:FOG request: DELETE /task/cancel {"id": 582385}
2023-11-04T04:12:07.489 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi097
2023-11-04T04:12:07.490 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-04T04:12:07.618 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=254b899653464289ad41f8e5b04794af
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi097
2023-11-04T04:12:07.621 INFO:teuthology.nuke:Checking targets against current locks
2023-11-04T04:12:07.684 DEBUG:teuthology.nuke:shortname: smithi097
2023-11-04T04:12:07.684 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-04T04:12:07.704 DEBUG:teuthology.nuke:shortname: smithi132
2023-11-04T04:12:07.704 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-04T04:12:07.708 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amaredia-2023-11-03_18:45:35-rgw-wip-rgw-swift-preserve-dash-underscore-in-container-metadata-distro-default-smithi/7445964', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-04 04:07:40.334460', 'locked_by': 'scheduled_amaredia@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}
2023-11-04T04:12:07.730 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amaredia-2023-11-03_18:45:35-rgw-wip-rgw-swift-preserve-dash-underscore-in-container-metadata-distro-default-smithi/7445964', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-04 04:07:40.336111', 'locked_by': 'scheduled_amaredia@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}
2023-11-04T04:12:07.758 INFO:teuthology.orchestra.console.smithi097:Power off
2023-11-04T04:12:07.758 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:12:07.793 INFO:teuthology.orchestra.console.smithi132:Power off
2023-11-04T04:12:07.794 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:12:07.816 DEBUG:teuthology.orchestra.console.smithi132:power off output: Chassis Power Control: Down/Off
2023-11-04T04:12:07.817 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:12:07.845 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is on
2023-11-04T04:12:11.849 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:12:11.958 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is on
2023-11-04T04:12:15.962 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:12:16.072 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is on
2023-11-04T04:12:20.074 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:12:20.185 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is off
2023-11-04T04:12:20.286 INFO:teuthology.orchestra.console.smithi132:Power off completed
2023-11-04T04:12:20.445 INFO:teuthology.lock.ops:unlocked: smithi132.front.sepia.ceph.com
2023-11-04T04:12:27.825 DEBUG:teuthology.orchestra.console.smithi097:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:12:27.826 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:12:47.892 DEBUG:teuthology.orchestra.console.smithi097:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:12:47.893 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-04T04:12:48.176 WARNING:teuthology.provision.fog.smithi132:timed out
2023-11-04T04:12:54.183 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:13:07.960 DEBUG:teuthology.orchestra.console.smithi097:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-11-04T04:13:07.961 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:13:08.970 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:13:12.702 WARNING:teuthology.provision.fog.smithi132:[Errno None] Unable to connect to port 22 on 172.21.15.132
2023-11-04T04:13:12.972 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:13:14.081 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:13:18.086 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:13:18.706 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:13:19.196 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:13:21.790 WARNING:teuthology.provision.fog.smithi132:[Errno None] Unable to connect to port 22 on 172.21.15.132
2023-11-04T04:13:23.198 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:13:24.306 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:13:27.795 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:13:28.308 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:13:29.418 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:13:30.878 WARNING:teuthology.provision.fog.smithi132:[Errno None] Unable to connect to port 22 on 172.21.15.132
2023-11-04T04:13:33.422 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:13:34.534 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:13:36.883 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:13:38.538 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:13:39.650 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:13:39.966 WARNING:teuthology.provision.fog.smithi132:[Errno None] Unable to connect to port 22 on 172.21.15.132
2023-11-04T04:13:43.654 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:13:44.764 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:13:45.969 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:13:48.767 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:13:49.054 WARNING:teuthology.provision.fog.smithi132:[Errno None] Unable to connect to port 22 on 172.21.15.132
2023-11-04T04:13:49.876 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:13:53.878 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:13:54.989 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:13:55.057 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:13:58.141 WARNING:teuthology.provision.fog.smithi132:[Errno None] Unable to connect to port 22 on 172.21.15.132
2023-11-04T04:13:58.994 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:14:00.103 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:14:04.106 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:14:04.144 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:14:05.217 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:14:07.201 WARNING:teuthology.provision.fog.smithi132:[Errno None] Unable to connect to port 22 on 172.21.15.132
2023-11-04T04:14:09.218 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:14:10.328 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:14:13.206 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-04T04:14:14.330 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:14:15.441 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:14:19.326 WARNING:teuthology.provision.fog.smithi132:[Errno None] Unable to connect to port 22 on 172.21.15.132
2023-11-04T04:14:19.442 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-04T04:14:20.553 DEBUG:teuthology.orchestra.console.smithi097:check power output: 
2023-11-04T04:14:20.553 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-11-04T04:14:20.554 ERROR:teuthology.orchestra.console.smithi097:Failed to power off
2023-11-04T04:14:20.702 INFO:teuthology.lock.ops:unlocked: smithi097.front.sepia.ceph.com
2023-11-04T04:14:20.939 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi097