2023-07-19T13:30:10.624 INFO:root:teuthology version: 0.0.1.dev153+g79491cb
2023-07-19T13:30:10.625 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-19T13:30:10.625 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-19T13:30:10.797 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f81501c95e0>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f814fb18940>, signals=[15])
2023-07-19T13:30:10.798 INFO:teuthology.lock.ops:Start node 'smithi043.front.sepia.ceph.com' reimaging
2023-07-19T13:30:10.798 INFO:teuthology.lock.ops:Updating [smithi043.front.sepia.ceph.com]: reset os type and version on server
2023-07-19T13:30:10.799 INFO:teuthology.lock.ops:Updating smithi043.front.sepia.ceph.com on lock server
2023-07-19T13:30:10.824 INFO:teuthology.lock.ops:Start node 'smithi199.front.sepia.ceph.com' reimaging
2023-07-19T13:30:10.825 INFO:teuthology.lock.ops:Updating [smithi199.front.sepia.ceph.com]: reset os type and version on server
2023-07-19T13:30:10.825 INFO:teuthology.lock.ops:Updating smithi199.front.sepia.ceph.com on lock server
2023-07-19T13:30:10.826 DEBUG:teuthology.provision.fog.smithi043:FOG request: GET /host {"name": "smithi043"}
2023-07-19T13:30:10.849 INFO:teuthology.lock.ops:Start node 'smithi203.front.sepia.ceph.com' reimaging
2023-07-19T13:30:10.850 INFO:teuthology.lock.ops:Updating [smithi203.front.sepia.ceph.com]: reset os type and version on server
2023-07-19T13:30:10.850 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server
2023-07-19T13:30:10.851 DEBUG:teuthology.provision.fog.smithi199:FOG request: GET /host {"name": "smithi199"}
2023-07-19T13:30:10.874 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /host {"name": "smithi203"}
2023-07-19T13:30:10.891 DEBUG:teuthology.provision.fog.smithi043:FOG request: GET /image {"name": "smithi_centos_8.stream"}
2023-07-19T13:30:10.925 DEBUG:teuthology.provision.fog.smithi043:Requesting image smithi_centos_8.stream (ID 3)
2023-07-19T13:30:10.925 DEBUG:teuthology.provision.fog.smithi043:FOG request: PUT /host/148 {"imageID": 3}
2023-07-19T13:30:10.937 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /image {"name": "smithi_centos_8.stream"}
2023-07-19T13:30:10.939 DEBUG:teuthology.provision.fog.smithi199:FOG request: GET /image {"name": "smithi_centos_8.stream"}
2023-07-19T13:30:10.970 INFO:teuthology.provision.fog.smithi043:Scheduling deploy of centos 8.stream
2023-07-19T13:30:10.971 DEBUG:teuthology.provision.fog.smithi043:FOG request: GET /task/active None
2023-07-19T13:30:10.973 DEBUG:teuthology.provision.fog.smithi199:Requesting image smithi_centos_8.stream (ID 3)
2023-07-19T13:30:10.973 DEBUG:teuthology.provision.fog.smithi199:FOG request: PUT /host/304 {"imageID": 3}
2023-07-19T13:30:10.980 DEBUG:teuthology.provision.fog.smithi203:Requesting image smithi_centos_8.stream (ID 3)
2023-07-19T13:30:10.981 DEBUG:teuthology.provision.fog.smithi203:FOG request: PUT /host/307 {"imageID": 3}
2023-07-19T13:30:11.020 INFO:teuthology.provision.fog.smithi203:Scheduling deploy of centos 8.stream
2023-07-19T13:30:11.021 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-19T13:30:11.026 INFO:teuthology.provision.fog.smithi199:Scheduling deploy of centos 8.stream
2023-07-19T13:30:11.026 DEBUG:teuthology.provision.fog.smithi199:FOG request: GET /task/active None
2023-07-19T13:30:11.133 DEBUG:teuthology.provision.fog.smithi043:FOG request: GET /tasktype {"name": "deploy"}
2023-07-19T13:30:11.143 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /tasktype {"name": "deploy"}
2023-07-19T13:30:11.149 DEBUG:teuthology.provision.fog.smithi199:FOG request: GET /tasktype {"name": "deploy"}
2023-07-19T13:30:11.165 DEBUG:teuthology.provision.fog.smithi043:FOG request: POST /host/148/task {"taskTypeID": 1}
2023-07-19T13:30:11.183 DEBUG:teuthology.provision.fog.smithi203:FOG request: POST /host/307/task {"taskTypeID": 1}
2023-07-19T13:30:11.189 DEBUG:teuthology.provision.fog.smithi199:FOG request: POST /host/304/task {"taskTypeID": 1}
2023-07-19T13:30:11.212 DEBUG:teuthology.provision.fog.smithi043:FOG request: GET /task/active None
2023-07-19T13:30:11.244 DEBUG:teuthology.provision.fog.smithi199:FOG request: GET /task/active None
2023-07-19T13:30:11.246 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-19T13:30:11.461 INFO:teuthology.orchestra.console:Power off smithi043
2023-07-19T13:30:11.462 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-19T13:30:11.478 INFO:teuthology.orchestra.console:Power off smithi203
2023-07-19T13:30:11.479 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-19T13:30:11.494 INFO:teuthology.orchestra.console:Power off smithi199
2023-07-19T13:30:11.494 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-19T13:30:11.502 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:30:11.509 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:30:15.524 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:30:15.534 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:30:19.640 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:30:19.648 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:30:19.851 INFO:teuthology.orchestra.console:Power off for smithi043 completed
2023-07-19T13:30:19.952 INFO:teuthology.orchestra.console:Power on smithi043
2023-07-19T13:30:19.952 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-19T13:30:19.972 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:30:23.760 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:30:23.970 INFO:teuthology.orchestra.console:Power off for smithi203 completed
2023-07-19T13:30:23.993 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:30:24.072 INFO:teuthology.orchestra.console:Power on smithi203
2023-07-19T13:30:24.073 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-19T13:30:24.094 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:30:28.109 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:30:28.119 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:30:28.321 INFO:teuthology.orchestra.console:Power on for smithi043 completed
2023-07-19T13:30:28.329 INFO:teuthology.orchestra.console:Power on for smithi203 completed
2023-07-19T13:30:28.422 INFO:teuthology.provision.fog.smithi043:Waiting for deploy to finish
2023-07-19T13:30:28.422 DEBUG:teuthology.provision.fog.smithi043:FOG request: GET /task/active None
2023-07-19T13:30:28.431 INFO:teuthology.provision.fog.smithi203:Waiting for deploy to finish
2023-07-19T13:30:28.431 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-19T13:30:31.565 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-19T13:30:43.678 DEBUG:teuthology.provision.fog.smithi043:FOG request: GET /task/active None
2023-07-19T13:30:43.691 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-19T13:30:51.636 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-19T13:30:58.928 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-19T13:30:58.931 DEBUG:teuthology.provision.fog.smithi043:FOG request: GET /task/active None
2023-07-19T13:31:11.705 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:31:14.157 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-19T13:31:14.169 DEBUG:teuthology.provision.fog.smithi043:FOG request: GET /task/active None
2023-07-19T13:31:16.716 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:31:21.830 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:31:26.940 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:31:29.444 DEBUG:teuthology.provision.fog.smithi043:FOG request: GET /task/active None
2023-07-19T13:31:29.452 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-19T13:31:32.052 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:31:37.166 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:31:42.279 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:31:44.919 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-19T13:31:44.935 DEBUG:teuthology.provision.fog.smithi043:FOG request: GET /task/active None
2023-07-19T13:31:47.391 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:31:52.504 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:31:57.618 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:32:00.439 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-19T13:32:00.481 DEBUG:teuthology.provision.fog.smithi043:FOG request: GET /task/active None
2023-07-19T13:32:02.730 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:32:07.840 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:32:12.954 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:32:15.995 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-19T13:32:16.033 DEBUG:teuthology.provision.fog.smithi043:FOG request: GET /task/active None
2023-07-19T13:32:18.066 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:32:23.181 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:32:24.290 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-07-19T13:32:24.291 ERROR:teuthology.orchestra.console:Failed to power off smithi199
2023-07-19T13:32:24.292 INFO:teuthology.orchestra.console:Power on smithi199
2023-07-19T13:32:24.292 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-19T13:32:32.820 DEBUG:teuthology.provision.fog.smithi043:FOG request: GET /task/active None
2023-07-19T13:32:32.837 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-19T13:32:44.363 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-19T13:32:48.464 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-19T13:32:48.473 DEBUG:teuthology.provision.fog.smithi043:FOG request: GET /task/active None
2023-07-19T13:33:04.438 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-19T13:33:04.694 DEBUG:teuthology.provision.fog.smithi043:FOG request: GET /task/active None
2023-07-19T13:33:04.703 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-19T13:33:22.462 DEBUG:teuthology.provision.fog.smithi043:FOG request: GET /task/active None
2023-07-19T13:33:22.510 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-19T13:33:24.511 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:33:29.521 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:33:34.632 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:33:39.572 DEBUG:teuthology.provision.fog.smithi043:FOG request: GET /task/active None
2023-07-19T13:33:39.582 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-19T13:33:39.742 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:33:44.856 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:33:49.970 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:33:55.083 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:33:56.967 DEBUG:teuthology.provision.fog.smithi043:FOG request: GET /task/active None
2023-07-19T13:33:57.042 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-19T13:34:00.198 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:34:05.313 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:34:10.427 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:34:15.252 DEBUG:teuthology.provision.fog.smithi043:FOG request: GET /task/active None
2023-07-19T13:34:15.349 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-19T13:34:15.537 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:34:20.652 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:34:25.766 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:34:30.879 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:34:31.578 DEBUG:teuthology.provision.fog.smithi043:FOG request: GET /task/active None
2023-07-19T13:34:31.649 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-19T13:34:35.993 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:34:37.103 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-07-19T13:34:37.103 DEBUG:teuthology.provision.fog.smithi199:Canceling deploy task with ID 395032
2023-07-19T13:34:37.104 DEBUG:teuthology.provision.fog.smithi199:FOG request: DELETE /task/cancel {"id": 395032}
2023-07-19T13:34:37.202 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/dispatcher/supervisor.py", line 227, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/console.py", line 243, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi199
2023-07-19T13:34:37.203 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-19T13:34:37.203 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-19T13:34:37.289 INFO:teuthology.nuke:Checking targets against current locks
2023-07-19T13:34:37.339 DEBUG:teuthology.nuke:shortname: smithi043
2023-07-19T13:34:37.340 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-19T13:34:37.367 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi043.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2023-07-19_13:28:53-orch:cephadm-wip-adk4-testing-2023-07-18-1038-distro-default-smithi/7343335', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-19 13:30:09.795537', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB9zYpH1X+3HSZ89peBA+9XJSxOMK2UW9+LxpXEtk0129af5XpATY2VCNs+jxdvJJewtYekmBA9qyWmby8BVtKg='}
2023-07-19T13:34:37.370 DEBUG:teuthology.nuke:shortname: smithi199
2023-07-19T13:34:37.370 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-19T13:34:37.386 DEBUG:teuthology.nuke:shortname: smithi203
2023-07-19T13:34:37.387 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-19T13:34:37.392 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi199.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2023-07-19_13:28:53-orch:cephadm-wip-adk4-testing-2023-07-18-1038-distro-default-smithi/7343335', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-19 13:30:09.794011', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuulMfG3lA1Lz4G6qLV3EqnpZrZKHM9iMapQjbaB+lbmCHxrb49ls73bIjWd5/A6/5+PDEo2aLmmzmRLKZtPQelTL3goFuwJI5aYnV97fBMpPlfDW1UYLmYwiKFgcMXs9/iykpOgy861fIjUkRSwtNrnE1WW3TMwsxYn08IPfll3R07tCkwrKLSX8uShjJmjYn921BAJ9AVRiBFYESY7BqyenXBThb0OUHYiVf8FQOqfAffvqIhz3ti9F0hGG6eeQz6v7e0HILPoNDvRBBW8fI5LG7PRDl4J4IM+6UBwAd7FGW45RHa+0xGKe/Ao8WMryicsq9qmWFsVdGSYEMasQXsnz9U1rFcvbxBSOo7mWmbnywfFcqz2s6jzJNMnxchhYjuQin7u0svXEoogFa2S6casNUp5w+R5RmYtdcEz16ZxI3p5NBUBN7F2DDLpTur5jgaVHKK40uIaBXoKPQjQXU7a/FacJkvfzQs7Ky12Y4JfEvceu432+0HblAfGs7yRU='}
2023-07-19T13:34:37.408 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2023-07-19_13:28:53-orch:cephadm-wip-adk4-testing-2023-07-18-1038-distro-default-smithi/7343335', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-19 13:30:09.796770', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='}
2023-07-19T13:34:37.417 INFO:teuthology.orchestra.console:Power off smithi043
2023-07-19T13:34:37.418 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-19T13:34:37.441 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:34:37.461 INFO:teuthology.orchestra.console:Power off smithi199
2023-07-19T13:34:37.462 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-19T13:34:37.498 INFO:teuthology.orchestra.console:Power off smithi203
2023-07-19T13:34:37.499 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-19T13:34:37.518 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:34:41.472 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:34:41.538 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:34:45.587 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:34:45.651 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:34:45.799 INFO:teuthology.orchestra.console:Power off for smithi043 completed
2023-07-19T13:34:45.952 INFO:teuthology.lock.ops:unlocked: smithi043.front.sepia.ceph.com
2023-07-19T13:34:47.729 DEBUG:teuthology.provision.fog.smithi043:FOG request: GET /task/active None
2023-07-19T13:34:47.733 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-19T13:34:49.761 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:34:49.971 INFO:teuthology.orchestra.console:Power off for smithi203 completed
2023-07-19T13:34:50.127 INFO:teuthology.lock.ops:unlocked: smithi203.front.sepia.ceph.com
2023-07-19T13:34:57.535 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-19T13:35:03.772 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-19T13:35:03.925 DEBUG:teuthology.provision.fog.smithi043:FOG request: GET /task/active None
2023-07-19T13:35:04.579 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-19T13:35:07.632 WARNING:teuthology.provision.fog.smithi043:[Errno None] Unable to connect to port 22 on 172.21.15.43
2023-07-19T13:35:13.639 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-19T13:35:16.720 WARNING:teuthology.provision.fog.smithi043:[Errno None] Unable to connect to port 22 on 172.21.15.43
2023-07-19T13:35:17.608 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-19T13:35:19.542 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-19T13:35:22.724 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-19T13:35:25.776 WARNING:teuthology.provision.fog.smithi043:[Errno None] Unable to connect to port 22 on 172.21.15.43
2023-07-19T13:35:31.783 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-19T13:35:34.832 WARNING:teuthology.provision.fog.smithi043:[Errno None] Unable to connect to port 22 on 172.21.15.43
2023-07-19T13:35:35.075 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-19T13:35:37.682 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:35:40.836 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-19T13:35:42.692 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:35:43.888 WARNING:teuthology.provision.fog.smithi043:[Errno None] Unable to connect to port 22 on 172.21.15.43
2023-07-19T13:35:47.807 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:35:49.891 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-19T13:35:50.676 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-19T13:35:52.918 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:35:52.944 WARNING:teuthology.provision.fog.smithi043:[Errno None] Unable to connect to port 22 on 172.21.15.43
2023-07-19T13:35:58.031 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:35:58.947 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-19T13:36:02.000 WARNING:teuthology.provision.fog.smithi043:[Errno None] Unable to connect to port 22 on 172.21.15.43
2023-07-19T13:36:03.142 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:36:06.300 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-19T13:36:08.002 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-19T13:36:08.252 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:36:13.365 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:36:18.479 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:36:21.789 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-19T13:36:23.592 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:36:28.701 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:36:33.812 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:36:37.215 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-19T13:36:38.923 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:36:39.652 ERROR:teuthology.orchestra.connection:Error authenticating with smithi043.front.sepia.ceph.com: Authentication failed.
2023-07-19T13:36:39.654 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-07-19T13:36:39.654 DEBUG:teuthology.orchestra.run.smithi043:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-07-19T13:36:39.655 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-19T13:36:40.659 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-19T13:36:41.662 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-19T13:36:42.665 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-19T13:36:43.669 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-19T13:36:44.033 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:36:44.673 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-19T13:36:45.677 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-19T13:36:46.681 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-19T13:36:47.685 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-19T13:36:48.689 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-19T13:36:49.143 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi199.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-19T13:36:49.694 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-19T13:36:50.253 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-07-19T13:36:50.253 ERROR:teuthology.orchestra.console:Failed to power off smithi199
2023-07-19T13:36:50.308 INFO:teuthology.lock.ops:unlocked: smithi199.front.sepia.ceph.com
2023-07-19T13:36:50.699 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-19T13:36:51.703 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-19T13:36:52.014 ERROR:teuthology.dispatcher.supervisor:Reimage failed 10 times ... marking machine down
2023-07-19T13:36:52.139 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/dispatcher/supervisor.py", line 57, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/dispatcher/supervisor.py", line 227, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/console.py", line 243, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi199