2023-07-19T22:33:56.487 INFO:root:teuthology version: 0.0.1.dev153+g79491cb
2023-07-19T22:33:56.489 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-19T22:33:56.489 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-19T22:34:26.198 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f569aa6e550>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f569a3bf820>, signals=[15])
2023-07-19T22:34:26.199 INFO:teuthology.lock.ops:Start node 'smithi032.front.sepia.ceph.com' reimaging
2023-07-19T22:34:26.199 INFO:teuthology.lock.ops:Updating [smithi032.front.sepia.ceph.com]: reset os type and version on server
2023-07-19T22:34:26.200 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server
2023-07-19T22:34:31.318 INFO:teuthology.lock.ops:Start node 'smithi064.front.sepia.ceph.com' reimaging
2023-07-19T22:34:31.318 INFO:teuthology.lock.ops:Updating [smithi064.front.sepia.ceph.com]: reset os type and version on server
2023-07-19T22:34:31.319 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server
2023-07-19T22:34:31.320 DEBUG:teuthology.provision.fog.smithi032:FOG request: GET /host {"name": "smithi032"}
2023-07-19T22:34:45.302 DEBUG:teuthology.provision.fog.smithi032:FOG request: GET /image {"name": "smithi_rhel_8.4"}
2023-07-19T22:34:50.310 INFO:teuthology.lock.ops:Start node 'smithi178.front.sepia.ceph.com' reimaging
2023-07-19T22:34:50.311 INFO:teuthology.lock.ops:Updating [smithi178.front.sepia.ceph.com]: reset os type and version on server
2023-07-19T22:34:50.311 INFO:teuthology.lock.ops:Updating smithi178.front.sepia.ceph.com on lock server
2023-07-19T22:34:50.312 DEBUG:teuthology.provision.fog.smithi064:FOG request: GET /host {"name": "smithi064"}
2023-07-19T22:35:00.340 DEBUG:teuthology.provision.fog.smithi178:FOG request: GET /host {"name": "smithi178"}
2023-07-19T22:35:05.348 DEBUG:teuthology.provision.fog.smithi032:Requesting image smithi_rhel_8.4 (ID 7)
2023-07-19T22:35:05.348 DEBUG:teuthology.provision.fog.smithi032:FOG request: PUT /host/137 {"imageID": 7}
2023-07-19T22:35:15.386 DEBUG:teuthology.provision.fog.smithi064:FOG request: GET /image {"name": "smithi_rhel_8.4"}
2023-07-19T22:35:20.393 INFO:teuthology.provision.fog.smithi032:Scheduling deploy of rhel 8.4
2023-07-19T22:35:20.394 DEBUG:teuthology.provision.fog.smithi032:FOG request: GET /task/active None
2023-07-19T22:35:30.431 DEBUG:teuthology.provision.fog.smithi178:FOG request: GET /image {"name": "smithi_rhel_8.4"}
2023-07-19T22:35:35.439 DEBUG:teuthology.provision.fog.smithi064:Requesting image smithi_rhel_8.4 (ID 7)
2023-07-19T22:35:35.439 DEBUG:teuthology.provision.fog.smithi064:FOG request: PUT /host/169 {"imageID": 7}
2023-07-19T22:35:35.441 DEBUG:teuthology.provision.fog.smithi032:FOG request: GET /tasktype {"name": "deploy"}
2023-07-19T22:35:40.492 DEBUG:teuthology.provision.fog.smithi178:Requesting image smithi_rhel_8.4 (ID 7)
2023-07-19T22:35:40.492 DEBUG:teuthology.provision.fog.smithi178:FOG request: PUT /host/283 {"imageID": 7}
2023-07-19T22:35:40.496 DEBUG:teuthology.provision.fog.smithi032:FOG request: POST /host/137/task {"taskTypeID": 1}
2023-07-19T22:35:40.499 INFO:teuthology.provision.fog.smithi064:Scheduling deploy of rhel 8.4
2023-07-19T22:35:40.500 DEBUG:teuthology.provision.fog.smithi064:FOG request: GET /task/active None
2023-07-19T22:35:40.550 INFO:teuthology.provision.fog.smithi178:Scheduling deploy of rhel 8.4
2023-07-19T22:35:40.550 DEBUG:teuthology.provision.fog.smithi178:FOG request: GET /task/active None
2023-07-19T22:35:40.558 DEBUG:teuthology.provision.fog.smithi064:FOG request: GET /tasktype {"name": "deploy"}
2023-07-19T22:35:40.568 DEBUG:teuthology.provision.fog.smithi032:FOG request: GET /task/active None
2023-07-19T22:35:40.597 DEBUG:teuthology.provision.fog.smithi064:FOG request: POST /host/169/task {"taskTypeID": 1}
2023-07-19T22:35:40.619 DEBUG:teuthology.provision.fog.smithi178:FOG request: GET /tasktype {"name": "deploy"}
2023-07-19T22:35:45.639 DEBUG:teuthology.provision.fog.smithi178:FOG request: POST /host/283/task {"taskTypeID": 1}
2023-07-19T22:35:45.641 DEBUG:teuthology.provision.fog.smithi064:FOG request: GET /task/active None
2023-07-19T22:35:55.684 INFO:teuthology.orchestra.console:Power off smithi032
2023-07-19T22:35:55.685 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-19T22:35:55.704 DEBUG:teuthology.provision.fog.smithi178:FOG request: GET /task/active None
2023-07-19T22:36:01.466 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:01.507 INFO:teuthology.orchestra.console:Power off smithi064
2023-07-19T22:36:01.508 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-19T22:36:01.531 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:01.579 INFO:teuthology.orchestra.console:Power off smithi178
2023-07-19T22:36:01.580 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-19T22:36:01.604 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:05.492 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:05.559 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:05.633 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:10.605 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:10.670 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:10.745 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:15.717 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:15.782 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:15.857 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:15.994 INFO:teuthology.orchestra.console:Power off for smithi064 completed
2023-07-19T22:36:16.068 INFO:teuthology.orchestra.console:Power off for smithi178 completed
2023-07-19T22:36:16.096 INFO:teuthology.orchestra.console:Power on smithi064
2023-07-19T22:36:16.096 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-19T22:36:16.117 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:16.170 INFO:teuthology.orchestra.console:Power on smithi178
2023-07-19T22:36:16.171 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-19T22:36:16.193 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:20.144 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:20.217 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:20.829 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:24.256 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:24.332 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:25.940 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:29.367 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:29.443 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:31.052 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:34.480 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:34.555 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:36.165 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:36.376 INFO:teuthology.orchestra.console:Power off for smithi032 completed
2023-07-19T22:36:36.477 INFO:teuthology.orchestra.console:Power on smithi032
2023-07-19T22:36:36.477 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-19T22:36:36.501 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:39.594 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:39.667 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:40.525 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:44.707 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:44.778 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:44.989 INFO:teuthology.orchestra.console:Power on for smithi178 completed
2023-07-19T22:36:45.090 INFO:teuthology.provision.fog.smithi178:Waiting for deploy to finish
2023-07-19T22:36:45.090 DEBUG:teuthology.provision.fog.smithi178:FOG request: GET /task/active None
2023-07-19T22:36:45.163 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-19T22:36:45.636 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:48.241 WARNING:teuthology.provision.fog.smithi178:[Errno None] Unable to connect to port 22 on 172.21.15.178
2023-07-19T22:36:49.818 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:50.029 INFO:teuthology.orchestra.console:Power on for smithi064 completed
2023-07-19T22:36:50.130 INFO:teuthology.provision.fog.smithi064:Waiting for deploy to finish
2023-07-19T22:36:50.131 DEBUG:teuthology.provision.fog.smithi064:FOG request: GET /task/active None
2023-07-19T22:36:50.192 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-19T22:36:50.747 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:53.264 WARNING:teuthology.provision.fog.smithi064:[Errno None] Unable to connect to port 22 on 172.21.15.64
2023-07-19T22:36:54.244 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-19T22:36:55.861 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:36:57.296 WARNING:teuthology.provision.fog.smithi178:[Errno None] Unable to connect to port 22 on 172.21.15.178
2023-07-19T22:36:59.266 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-19T22:37:00.974 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:37:02.320 WARNING:teuthology.provision.fog.smithi064:[Errno None] Unable to connect to port 22 on 172.21.15.64
2023-07-19T22:37:03.300 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-19T22:37:08.305 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:37:08.322 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-19T22:37:11.376 WARNING:teuthology.provision.fog.smithi178:[Errno None] Unable to connect to port 22 on 172.21.15.178
2023-07-19T22:37:11.377 WARNING:teuthology.provision.fog.smithi064:[Errno None] Unable to connect to port 22 on 172.21.15.64
2023-07-19T22:37:13.417 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:37:17.381 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-19T22:37:22.389 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-19T22:37:27.393 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:37:27.401 WARNING:teuthology.provision.fog.smithi178:[Errno None] Unable to connect to port 22 on 172.21.15.178
2023-07-19T22:37:30.448 WARNING:teuthology.provision.fog.smithi064:[Errno None] Unable to connect to port 22 on 172.21.15.64
2023-07-19T22:37:32.503 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:37:33.404 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-19T22:37:36.450 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-19T22:37:41.454 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:37:41.462 WARNING:teuthology.provision.fog.smithi178:[Errno None] Unable to connect to port 22 on 172.21.15.178
2023-07-19T22:37:46.564 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:37:47.465 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-19T22:37:51.674 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:37:56.786 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:37:59.792 WARNING:teuthology.provision.fog.smithi064:[Errno None] Unable to connect to port 22 on 172.21.15.64
2023-07-19T22:38:01.899 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:38:03.011 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-07-19T22:38:03.012 DEBUG:teuthology.provision.fog.smithi032:Canceling deploy task with ID 396512
2023-07-19T22:38:03.012 DEBUG:teuthology.provision.fog.smithi032:FOG request: DELETE /task/cancel {"id": 396512}
2023-07-19T22:38:08.021 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-19T22:38:13.025 WARNING:teuthology.provision.fog.smithi178:[Errno None] Unable to connect to port 22 on 172.21.15.178
2023-07-19T22:38:13.085 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/dispatcher/supervisor.py", line 227, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/console.py", line 243, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi032
2023-07-19T22:38:13.086 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-19T22:38:13.086 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-19T22:38:16.526 ERROR:teuthology.orchestra.connection:Error authenticating with smithi064.front.sepia.ceph.com: Authentication failed.
2023-07-19T22:38:16.528 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-07-19T22:38:16.529 DEBUG:teuthology.orchestra.run.smithi064:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-07-19T22:38:16.531 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-19T22:38:16.650 INFO:teuthology.nuke:Checking targets against current locks
2023-07-19T22:38:16.704 DEBUG:teuthology.nuke:shortname: smithi032
2023-07-19T22:38:16.705 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-19T22:38:16.729 DEBUG:teuthology.nuke:shortname: smithi064
2023-07-19T22:38:16.729 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-19T22:38:16.737 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-19_14:40:49-rbd-wip-yuri11-testing-2023-07-18-0927-pacific-distro-default-smithi/7343911', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-19 22:33:55.628193', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGIEb1iGeAkvUZirLl+TOES+N1AA1yVI7z41kEXXmrrsuR7158HqkjO5nqwvcqbN1mMsVOp12PO17ixN5zy4ciKMdWbIDuL+bFF8CKVis5Ly2ZiS8xCN9zZq967zJdCm3z2Ei8KaWoWk+Bw8G0H1lgENdKKB19Hd6SjNta6jojaj8LUtkmiNN0LIHSYePK6bg0tB2S4TfzBtUuANqmgq+UUtyeSPkEDr8InltSRhmgzUl70lqmUY5V+wDEvVsUCKUYgyiNWbbcdocDLjm1S/OjWpvn33mjwpHd9RphxnZ2dO5wKNIWNFB8uOMAOI5gewIFJiLJD5SIOAvVRVux7b/TRvPNq6GA1H5dJ5aX7Imd3xKWyJiUK/EJO78KmZjTid6lE1k1sExedQFT4i3i94h4PU3pHW4dmTEi9iEoc3XLT0cqJexwnz2zSJx2DsSkrfvu4AgBlBQmq33Axv9tlTG2YHmjaw0D8jte4GnznM853lvuTy4SNM3B59JIo/p7m7c='}
2023-07-19T22:38:16.748 DEBUG:teuthology.nuke:shortname: smithi178
2023-07-19T22:38:16.749 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-19T22:38:16.755 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-19_14:40:49-rbd-wip-yuri11-testing-2023-07-18-0927-pacific-distro-default-smithi/7343911', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-19 22:33:55.631799', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCcFOS18Cb2IUjAIvmaDyN2mOa2fGnwDclDd4b+ZLQY6WCj5MzywQjz+AO1NVx1oanb6PRhUTsmAnUC0jxX6EdAf03JH/4KwO0EYqphUi3IgNcqhZacui2vKyOXho1fDc5/UdYiA+l3WSFiqaZn6a2NqoZs7xPPCSc3puWFbAp4jNt7nFDgMH4EfW/eH8u3CxnxpLUDa6GU1+QaaAQePzzoMjRiRLx/5u/cSRRzMKqiLxOpjP3GQIlJFnLsKlr/pBkCxb5kacxzs4smdHbE7FwUzsB+2ynfcMUlBDconoGm1rWzptly2M5iaqHjfFh6VLaoBHMXioKHgTnl1d79BZkJnGz7b0gA3gOjp17bfI6FmjsdlCwRg/9qvfO0ootHZV0K9FdAL8AMxu1WaPXIwp0wCAEdfjsTejBlc/LJHUzgwjZnr1Ysm4sX94rFJZT/g54l+CLwdN/1E0w7xHneV4KxwC0nc5f30bz7dN52ddTX3OQcmjeUWFWZd1LDgPV5G8c='}
2023-07-19T22:38:16.772 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-19_14:40:49-rbd-wip-yuri11-testing-2023-07-18-0927-pacific-distro-default-smithi/7343911', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-19 22:33:55.630449', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzBXKlm/fDuKb8/ogsl/rj9yCJHqcSd0m2pN5HlCIrzuCWm0zpnGr6IpS4qlll1fRkx29OXRO2BX1D3Na0ujEUCx0QQFrlow4RrcJKmMI8ywTvl4XkmKMwnr4gJa5/jimVvE+mKPDkuPzYVnO9QuZBZfjT/V07/woOeyUO8RnXXQL0rdhx3R3zDCJglosYitoPssqlM2rVUTaWi+KKS8IbQbhu6Ozw1CJ0qCkgqP25PPht8o/jHz4NAxql1zEiozhW1GYnWZqVPlfCkpPIEhFNCmV/S4iKEd+yV0c6F5Xn6nBV0f34wERFnhi+5nEBeYnXYjinkNNd9KFMgsIAyOGpcjHTjSXQXlKyZC1pNmLkzpPUTd2J6H0TpPSA3zyTgpBL/GPvjFxSv8girj9BbM5dM+trD/6ORZSQbUJYKOjzXQxiNUDcz7D+WQjPAt+T3pbqgJZno3xsEES3m+CZzVoKNsv9jHSnSC4n3/o7uhQC+gRZz/GqDYVyyZ7MelI6yBk='}
2023-07-19T22:38:16.789 INFO:teuthology.orchestra.console:Power off smithi032
2023-07-19T22:38:16.790 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-19T22:38:16.814 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:38:16.831 INFO:teuthology.orchestra.console:Power off smithi064
2023-07-19T22:38:16.832 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-19T22:38:16.848 INFO:teuthology.orchestra.console:Power off smithi178
2023-07-19T22:38:16.849 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-19T22:38:16.859 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:38:16.881 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:38:17.545 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-19T22:38:18.551 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-19T22:38:19.029 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-19T22:38:24.037 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:38:24.047 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:38:24.059 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:38:24.069 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-19T22:38:24.186 ERROR:teuthology.orchestra.connection:Error authenticating with smithi178.front.sepia.ceph.com: Authentication failed.
2023-07-19T22:38:24.187 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-07-19T22:38:24.187 DEBUG:teuthology.orchestra.run.smithi178:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-07-19T22:38:24.189 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-19T22:38:25.078 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-19T22:38:28.151 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:38:28.164 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:38:28.174 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T22:38:28.363 INFO:teuthology.orchestra.console:Power off for smithi032 completed
2023-07-19T22:38:28.377 INFO:teuthology.orchestra.console:Power off for smithi064 completed
2023-07-19T22:38:28.387 INFO:teuthology.orchestra.console:Power off for smithi178 completed
2023-07-19T22:38:28.521 INFO:teuthology.lock.ops:unlocked: smithi032.front.sepia.ceph.com
2023-07-19T22:38:28.536 INFO:teuthology.lock.ops:unlocked: smithi064.front.sepia.ceph.com
2023-07-19T22:38:28.546 INFO:teuthology.lock.ops:unlocked: smithi178.front.sepia.ceph.com
2023-07-19T22:38:28.941 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/dispatcher/supervisor.py", line 57, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/dispatcher/supervisor.py", line 227, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/console.py", line 243, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi032