2023-07-12T01:25:00.401 INFO:root:teuthology version: 0.0.1.dev140+gb53f236
2023-07-12T01:25:00.403 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-12T01:25:00.403 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-12T01:25:00.655 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7face5db35e0>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7face571b790>, signals=[15])
2023-07-12T01:25:00.656 INFO:teuthology.lock.ops:Start node 'smithi042.front.sepia.ceph.com' reimaging
2023-07-12T01:25:00.657 INFO:teuthology.lock.ops:Updating [smithi042.front.sepia.ceph.com]: reset os type and version on server
2023-07-12T01:25:00.657 INFO:teuthology.lock.ops:Updating smithi042.front.sepia.ceph.com on lock server
2023-07-12T01:25:00.677 INFO:teuthology.lock.ops:Start node 'smithi069.front.sepia.ceph.com' reimaging
2023-07-12T01:25:00.678 INFO:teuthology.lock.ops:Updating [smithi069.front.sepia.ceph.com]: reset os type and version on server
2023-07-12T01:25:00.678 INFO:teuthology.lock.ops:Updating smithi069.front.sepia.ceph.com on lock server
2023-07-12T01:25:00.679 DEBUG:teuthology.provision.fog.smithi042:FOG request: GET /host {"name": "smithi042"}
2023-07-12T01:25:00.703 INFO:teuthology.lock.ops:Start node 'smithi123.front.sepia.ceph.com' reimaging
2023-07-12T01:25:00.703 INFO:teuthology.lock.ops:Updating [smithi123.front.sepia.ceph.com]: reset os type and version on server
2023-07-12T01:25:00.704 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server
2023-07-12T01:25:00.706 DEBUG:teuthology.provision.fog.smithi069:FOG request: GET /host {"name": "smithi069"}
2023-07-12T01:25:00.727 INFO:teuthology.lock.ops:Start node 'smithi137.front.sepia.ceph.com' reimaging
2023-07-12T01:25:00.727 INFO:teuthology.lock.ops:Updating [smithi137.front.sepia.ceph.com]: reset os type and version on server
2023-07-12T01:25:00.728 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server
2023-07-12T01:25:00.729 DEBUG:teuthology.provision.fog.smithi123:FOG request: GET /host {"name": "smithi123"}
2023-07-12T01:25:00.748 DEBUG:teuthology.provision.fog.smithi042:FOG request: GET /image {"name": "smithi_ubuntu_22.04"}
2023-07-12T01:25:00.752 INFO:teuthology.lock.ops:Start node 'smithi153.front.sepia.ceph.com' reimaging
2023-07-12T01:25:00.753 INFO:teuthology.lock.ops:Updating [smithi153.front.sepia.ceph.com]: reset os type and version on server
2023-07-12T01:25:00.753 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server
2023-07-12T01:25:00.754 DEBUG:teuthology.provision.fog.smithi137:FOG request: GET /host {"name": "smithi137"}
2023-07-12T01:25:00.777 DEBUG:teuthology.provision.fog.smithi153:FOG request: GET /host {"name": "smithi153"}
2023-07-12T01:25:00.780 DEBUG:teuthology.provision.fog.smithi069:FOG request: GET /image {"name": "smithi_ubuntu_22.04"}
2023-07-12T01:25:00.795 DEBUG:teuthology.provision.fog.smithi042:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-07-12T01:25:00.795 DEBUG:teuthology.provision.fog.smithi042:FOG request: PUT /host/147 {"imageID": 6}
2023-07-12T01:25:00.803 DEBUG:teuthology.provision.fog.smithi123:FOG request: GET /image {"name": "smithi_ubuntu_22.04"}
2023-07-12T01:25:00.817 DEBUG:teuthology.provision.fog.smithi069:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-07-12T01:25:00.818 DEBUG:teuthology.provision.fog.smithi069:FOG request: PUT /host/174 {"imageID": 6}
2023-07-12T01:25:00.824 DEBUG:teuthology.provision.fog.smithi137:FOG request: GET /image {"name": "smithi_ubuntu_22.04"}
2023-07-12T01:25:00.838 DEBUG:teuthology.provision.fog.smithi123:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-07-12T01:25:00.838 DEBUG:teuthology.provision.fog.smithi123:FOG request: PUT /host/228 {"imageID": 6}
2023-07-12T01:25:00.840 INFO:teuthology.provision.fog.smithi042:Scheduling deploy of ubuntu 22.04
2023-07-12T01:25:00.841 DEBUG:teuthology.provision.fog.smithi042:FOG request: GET /task/active None
2023-07-12T01:25:00.843 DEBUG:teuthology.provision.fog.smithi153:FOG request: GET /image {"name": "smithi_ubuntu_22.04"}
2023-07-12T01:25:00.858 INFO:teuthology.provision.fog.smithi069:Scheduling deploy of ubuntu 22.04
2023-07-12T01:25:00.859 DEBUG:teuthology.provision.fog.smithi069:FOG request: GET /task/active None
2023-07-12T01:25:00.861 DEBUG:teuthology.provision.fog.smithi137:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-07-12T01:25:00.861 DEBUG:teuthology.provision.fog.smithi137:FOG request: PUT /host/242 {"imageID": 6}
2023-07-12T01:25:00.888 DEBUG:teuthology.provision.fog.smithi153:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-07-12T01:25:00.889 DEBUG:teuthology.provision.fog.smithi153:FOG request: PUT /host/258 {"imageID": 6}
2023-07-12T01:25:00.892 INFO:teuthology.provision.fog.smithi123:Scheduling deploy of ubuntu 22.04
2023-07-12T01:25:00.892 DEBUG:teuthology.provision.fog.smithi123:FOG request: GET /task/active None
2023-07-12T01:25:00.899 INFO:teuthology.provision.fog.smithi137:Scheduling deploy of ubuntu 22.04
2023-07-12T01:25:00.900 DEBUG:teuthology.provision.fog.smithi137:FOG request: GET /task/active None
2023-07-12T01:25:00.933 INFO:teuthology.provision.fog.smithi153:Scheduling deploy of ubuntu 22.04
2023-07-12T01:25:00.933 DEBUG:teuthology.provision.fog.smithi153:FOG request: GET /task/active None
2023-07-12T01:25:00.943 DEBUG:teuthology.provision.fog.smithi042:FOG request: GET /tasktype {"name": "deploy"}
2023-07-12T01:25:00.959 DEBUG:teuthology.provision.fog.smithi069:Canceling deploy task with ID 378687
2023-07-12T01:25:00.960 DEBUG:teuthology.provision.fog.smithi069:FOG request: DELETE /task/cancel {"id": 378687}
2023-07-12T01:25:00.984 DEBUG:teuthology.provision.fog.smithi123:Canceling deploy task with ID 378685
2023-07-12T01:25:00.984 DEBUG:teuthology.provision.fog.smithi123:FOG request: DELETE /task/cancel {"id": 378685}
2023-07-12T01:25:00.988 DEBUG:teuthology.provision.fog.smithi042:FOG request: POST /host/147/task {"taskTypeID": 1}
2023-07-12T01:25:00.996 DEBUG:teuthology.provision.fog.smithi137:Canceling deploy task with ID 378686
2023-07-12T01:25:00.996 DEBUG:teuthology.provision.fog.smithi137:FOG request: DELETE /task/cancel {"id": 378686}
2023-07-12T01:25:01.003 DEBUG:teuthology.provision.fog.smithi069:FOG request: GET /tasktype {"name": "deploy"}
2023-07-12T01:25:01.018 DEBUG:teuthology.provision.fog.smithi153:FOG request: GET /tasktype {"name": "deploy"}
2023-07-12T01:25:01.035 DEBUG:teuthology.provision.fog.smithi123:FOG request: GET /tasktype {"name": "deploy"}
2023-07-12T01:25:01.037 DEBUG:teuthology.provision.fog.smithi137:FOG request: GET /tasktype {"name": "deploy"}
2023-07-12T01:25:01.042 DEBUG:teuthology.provision.fog.smithi069:FOG request: POST /host/174/task {"taskTypeID": 1}
2023-07-12T01:25:01.044 DEBUG:teuthology.provision.fog.smithi042:FOG request: GET /task/active None
2023-07-12T01:25:01.053 DEBUG:teuthology.provision.fog.smithi153:FOG request: POST /host/258/task {"taskTypeID": 1}
2023-07-12T01:25:01.081 DEBUG:teuthology.provision.fog.smithi137:FOG request: POST /host/242/task {"taskTypeID": 1}
2023-07-12T01:25:01.083 DEBUG:teuthology.provision.fog.smithi123:FOG request: POST /host/228/task {"taskTypeID": 1}
2023-07-12T01:25:01.101 DEBUG:teuthology.provision.fog.smithi069:FOG request: GET /task/active None
2023-07-12T01:25:01.104 DEBUG:teuthology.provision.fog.smithi153:FOG request: GET /task/active None
2023-07-12T01:25:01.144 DEBUG:teuthology.provision.fog.smithi137:FOG request: GET /task/active None
2023-07-12T01:25:01.146 DEBUG:teuthology.provision.fog.smithi123:FOG request: GET /task/active None
2023-07-12T01:25:01.165 INFO:teuthology.orchestra.console:Power off smithi042
2023-07-12T01:25:01.166 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-12T01:25:01.187 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:01.233 INFO:teuthology.orchestra.console:Power off smithi153
2023-07-12T01:25:01.234 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-12T01:25:01.255 INFO:teuthology.orchestra.console:Power off smithi069
2023-07-12T01:25:01.256 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-12T01:25:01.269 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:01.279 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:01.297 INFO:teuthology.orchestra.console:Power off smithi123
2023-07-12T01:25:01.298 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-12T01:25:01.314 INFO:teuthology.orchestra.console:Power off smithi137
2023-07-12T01:25:01.314 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-12T01:25:01.325 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:01.337 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:01.425 INFO:teuthology.orchestra.console:Power off for smithi069 completed
2023-07-12T01:25:01.459 INFO:teuthology.orchestra.console:Power off for smithi137 completed
2023-07-12T01:25:01.526 INFO:teuthology.orchestra.console:Power on smithi069
2023-07-12T01:25:01.526 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-12T01:25:01.546 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:01.561 INFO:teuthology.orchestra.console:Power on smithi137
2023-07-12T01:25:01.561 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-12T01:25:01.580 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:05.212 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:05.325 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:05.349 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:05.560 INFO:teuthology.orchestra.console:Power off for smithi123 completed
2023-07-12T01:25:05.571 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:05.603 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:05.661 INFO:teuthology.orchestra.console:Power on smithi123
2023-07-12T01:25:05.662 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-12T01:25:05.685 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:05.781 INFO:teuthology.orchestra.console:Power on for smithi069 completed
2023-07-12T01:25:05.882 INFO:teuthology.provision.fog.smithi069:Waiting for deploy to finish
2023-07-12T01:25:05.883 DEBUG:teuthology.provision.fog.smithi069:FOG request: GET /task/active None
2023-07-12T01:25:09.324 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:09.438 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:09.536 INFO:teuthology.orchestra.console:Power off for smithi042 completed
2023-07-12T01:25:09.637 INFO:teuthology.orchestra.console:Power on smithi042
2023-07-12T01:25:09.638 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-12T01:25:09.648 INFO:teuthology.orchestra.console:Power off for smithi153 completed
2023-07-12T01:25:09.659 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:09.706 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:09.715 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:09.750 INFO:teuthology.orchestra.console:Power on smithi153
2023-07-12T01:25:09.751 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-12T01:25:09.772 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:09.924 INFO:teuthology.orchestra.console:Power on for smithi137 completed
2023-07-12T01:25:10.025 INFO:teuthology.provision.fog.smithi137:Waiting for deploy to finish
2023-07-12T01:25:10.026 DEBUG:teuthology.provision.fog.smithi137:FOG request: GET /task/active None
2023-07-12T01:25:13.684 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:13.793 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:13.816 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:17.805 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:17.905 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:17.926 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:18.016 INFO:teuthology.orchestra.console:Power on for smithi042 completed
2023-07-12T01:25:18.116 INFO:teuthology.orchestra.console:Power on for smithi153 completed
2023-07-12T01:25:18.117 INFO:teuthology.provision.fog.smithi042:Waiting for deploy to finish
2023-07-12T01:25:18.117 DEBUG:teuthology.provision.fog.smithi042:FOG request: GET /task/active None
2023-07-12T01:25:18.218 INFO:teuthology.provision.fog.smithi153:Waiting for deploy to finish
2023-07-12T01:25:18.219 DEBUG:teuthology.provision.fog.smithi153:FOG request: GET /task/active None
2023-07-12T01:25:21.050 DEBUG:teuthology.provision.fog.smithi069:FOG request: GET /task/active None
2023-07-12T01:25:22.037 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:25.168 DEBUG:teuthology.provision.fog.smithi137:FOG request: GET /task/active None
2023-07-12T01:25:26.147 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:30.261 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:33.269 DEBUG:teuthology.provision.fog.smithi042:FOG request: GET /task/active None
2023-07-12T01:25:33.364 DEBUG:teuthology.provision.fog.smithi153:FOG request: GET /task/active None
2023-07-12T01:25:34.371 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:36.190 DEBUG:teuthology.provision.fog.smithi069:FOG request: GET /task/active None
2023-07-12T01:25:38.485 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:40.303 DEBUG:teuthology.provision.fog.smithi137:FOG request: GET /task/active None
2023-07-12T01:25:42.597 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:46.711 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:48.412 DEBUG:teuthology.provision.fog.smithi042:FOG request: GET /task/active None
2023-07-12T01:25:48.501 DEBUG:teuthology.provision.fog.smithi153:FOG request: GET /task/active None
2023-07-12T01:25:50.823 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:51.329 DEBUG:teuthology.provision.fog.smithi069:FOG request: GET /task/active None
2023-07-12T01:25:54.936 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:25:55.466 DEBUG:teuthology.provision.fog.smithi137:FOG request: GET /task/active None
2023-07-12T01:25:59.048 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:26:03.161 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:26:03.270 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-07-12T01:26:03.371 DEBUG:teuthology.provision.fog.smithi123:Canceling deploy task with ID 378710
2023-07-12T01:26:03.372 DEBUG:teuthology.provision.fog.smithi123:FOG request: DELETE /task/cancel {"id": 378710}
2023-07-12T01:26:03.426 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/dispatcher/supervisor.py", line 227, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/console.py", line 243, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi123
2023-07-12T01:26:03.426 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-12T01:26:03.427 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-12T01:26:03.507 INFO:teuthology.nuke:Checking targets against current locks
2023-07-12T01:26:03.550 DEBUG:teuthology.provision.fog.smithi042:FOG request: GET /task/active None
2023-07-12T01:26:03.556 DEBUG:teuthology.nuke:shortname: smithi042
2023-07-12T01:26:03.557 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-12T01:26:03.579 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi042.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334514', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-12 01:24:59.522124', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCEsPjBPX6guTOpInWYxLfRytWcp3dfjD3eQILSBnLcBMIkMqzTtYIvxViDOalLh5YSrMLUEhBR3D/5+q4pxd9OFysSvq0utKLRMWuvaBtrSLPrQk6EY37dZizWqt7KVMTJ0seoqHnOxfYTIdksUfXHDtK2KlNqrfBMN7RnDi6EShoRlprFIKBWl5SzB80WxHpaqY9pWeu/qCqCAJ+VMCbWzRownR52Fj9fsqI8EV6QG/DroiTwzL7ucd+7yGXmFnDBDcgBGDnOfpEL85MZCwIq9rtFIQKtF1Kst+wt5nZP7i/39TpIoC6USalMgfM1gL0I2nNl/9RtdyHkaCpFvAefHdDvMMLaMXoo7KPvkdMzxUgrGV3IUkHqTi78yjXbFgZ7wwr9TfOQCj8Kw/B6SSlB8QwyKAw8trRecsdkwQ5QzS9H9aoTcBjPZMlggdi6+UAQ7XjvPfxEGlBpUaTtHirNAXVu6fln4dO7N7Y2lHZNU+1ZYLAl4sUSXJ0Qbjh1gc='}
2023-07-12T01:26:03.588 DEBUG:teuthology.nuke:shortname: smithi069
2023-07-12T01:26:03.588 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-12T01:26:03.608 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334514', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-12 01:24:59.523185', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQnghTqeHX9mTHIGtXUyfxMmcUZjx3iIPyVE1cuA2ihIiH+Zi91namtNlRukU0lIULrwpAa/tguBNfmFSAxsZYMOsn31uPs/yGzYKgjDbWczLjNITK5lqmFwzsb4/2qQx4hiphe5vPpvbJMpGQ1aWWya4/Wl7u0q2LQCDvdZnEGlUDwHewCkzzNB5f6W6teIsTSpTGGc0YeKR1+TE6Xjzhne1DqBVahJp2WilcIQoS0C6C/+UD0Ao0ZupCysPSeZrzXM4smz+KDOxQVGfen/+qcTTFJ2kSRD5tZyYGqXtxHwDjIfcDwBkcBOrSBSjcqnJIF3gbXW3FvStIhtoGzMQBhFn5LRY2xOSGt51YYcWS/Gz2vWZGdptJ5g3gnGcemOa8Us1UYkqekQibL/iPMrcmKQslzV6F6DblJjjiIKVy9lCHrZJ76A4hCPwkzzleVuAWMrD1NUslbTlsrhfS8CcN5E0M6EspsyAnaCovQZ3JR1aeMdWDpkx+99iMNN/oUBM='}
2023-07-12T01:26:03.610 DEBUG:teuthology.nuke:shortname: smithi123
2023-07-12T01:26:03.611 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-12T01:26:03.627 INFO:teuthology.orchestra.console:Power off smithi042
2023-07-12T01:26:03.627 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-12T01:26:03.635 DEBUG:teuthology.provision.fog.smithi153:FOG request: GET /task/active None
2023-07-12T01:26:03.643 DEBUG:teuthology.nuke:shortname: smithi137
2023-07-12T01:26:03.643 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-12T01:26:03.644 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334514', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-12 01:24:59.519325', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}
2023-07-12T01:26:03.649 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:26:03.672 DEBUG:teuthology.nuke:shortname: smithi153
2023-07-12T01:26:03.672 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-12T01:26:03.676 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334514', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-12 01:24:59.521071', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDN5Qj6wMbogAjTs2JvVeikcUr7cziXluMgeOWqRhieNz2rdCq3YkcbJsxUWcIBvJk+AQKl4b99Ow+8HvEQaBevArKTIZnKWf6sq9dX1fTn0IIMaHCHqGEMumFIFUUIW5TtRFOzXfY64lXtqoyySpDDKdR44N3vCoMbI3ZBWHnGC+M5CVi6a0m+K0pAWELgDccrRwdfKP3mf3SAPwscpd3LzBtRmmpQqp3B3W39sshxkZWSSLBkPHeXghEiSm2dnBSfXi8Y3HTVB6WJ3VQFVfOFj86jQEayZTJdeiPjJpiuzetZ4QWVuNqhn7KiYS7V5OP9MG5U/rCaH2HNN4LpyGzCZG9USMqQWboNbamLMXN6JEdOPF0BySSZtAh+s2jQKKRf0853kYST7Rd+CWdBrfgP6495JB4aVRtPYjBqINW0hx3SrvEY0B9INXPTK/yrmVO5wIJUYTphJAz1baE9ZEpdJ8aGi2eJrT3UnPMmK1HiEu19Hz97VSpHQkspE+jlw9c='}
2023-07-12T01:26:03.681 INFO:teuthology.orchestra.console:Power off smithi069
2023-07-12T01:26:03.682 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-12T01:26:03.701 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:26:03.714 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-12_01:23:58-fs-wip-pdonnell-testing-20230710.195410-distro-default-smithi/7334514', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-12 01:24:59.524293', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='}
2023-07-12T01:26:03.722 INFO:teuthology.orchestra.console:Power off smithi123
2023-07-12T01:26:03.723 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-12T01:26:03.740 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:26:03.761 INFO:teuthology.orchestra.console:Power off smithi137
2023-07-12T01:26:03.761 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-12T01:26:03.797 INFO:teuthology.orchestra.console:Power off smithi153
2023-07-12T01:26:03.797 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-12T01:26:03.816 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:26:03.829 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:26:03.871 INFO:teuthology.orchestra.console:Power off for smithi123 completed
2023-07-12T01:26:04.094 INFO:teuthology.lock.ops:unlocked: smithi123.front.sepia.ceph.com
2023-07-12T01:26:06.471 DEBUG:teuthology.provision.fog.smithi069:FOG request: GET /task/active None
2023-07-12T01:26:07.678 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi042.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:26:07.732 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:26:07.838 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:26:07.860 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-12T01:26:07.888 INFO:teuthology.orchestra.console:Power off for smithi042 completed
2023-07-12T01:26:07.947 INFO:teuthology.orchestra.console:Power off for smithi069 completed
2023-07-12T01:26:08.046 INFO:teuthology.lock.ops:unlocked: smithi042.front.sepia.ceph.com
2023-07-12T01:26:08.048 INFO:teuthology.orchestra.console:Power off for smithi153 completed
2023-07-12T01:26:08.070 INFO:teuthology.orchestra.console:Power off for smithi137 completed
2023-07-12T01:26:08.096 INFO:teuthology.lock.ops:unlocked: smithi069.front.sepia.ceph.com
2023-07-12T01:26:08.196 INFO:teuthology.lock.ops:unlocked: smithi153.front.sepia.ceph.com
2023-07-12T01:26:08.213 INFO:teuthology.lock.ops:unlocked: smithi137.front.sepia.ceph.com
2023-07-12T01:26:08.879 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/dispatcher/supervisor.py", line 57, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/dispatcher/supervisor.py", line 227, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/console.py", line 243, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi123