2023-07-20T19:22:33.127 INFO:root:teuthology version: 0.0.1.dev153+g79491cb
2023-07-20T19:22:33.129 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-20T19:22:33.129 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-20T19:22:47.944 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f61bed15670>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f61be6c69d0>, signals=[15])
2023-07-20T19:22:47.946 INFO:teuthology.lock.ops:Start node 'smithi060.front.sepia.ceph.com' reimaging
2023-07-20T19:22:47.946 INFO:teuthology.lock.ops:Updating [smithi060.front.sepia.ceph.com]: reset os type and version on server
2023-07-20T19:22:47.947 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server
2023-07-20T19:22:47.965 INFO:teuthology.lock.ops:Start node 'smithi119.front.sepia.ceph.com' reimaging
2023-07-20T19:22:47.966 INFO:teuthology.lock.ops:Updating [smithi119.front.sepia.ceph.com]: reset os type and version on server
2023-07-20T19:22:47.966 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server
2023-07-20T19:22:47.968 DEBUG:teuthology.provision.fog.smithi060:FOG request: GET /host {"name": "smithi060"}
2023-07-20T19:22:54.413 INFO:teuthology.lock.ops:Start node 'smithi139.front.sepia.ceph.com' reimaging
2023-07-20T19:22:54.414 INFO:teuthology.lock.ops:Updating [smithi139.front.sepia.ceph.com]: reset os type and version on server
2023-07-20T19:22:54.414 INFO:teuthology.lock.ops:Updating smithi139.front.sepia.ceph.com on lock server
2023-07-20T19:22:54.415 DEBUG:teuthology.provision.fog.smithi119:FOG request: GET /host {"name": "smithi119"}
2023-07-20T19:23:02.249 DEBUG:teuthology.provision.fog.smithi139:FOG request: GET /host {"name": "smithi139"}
2023-07-20T19:23:02.278 DEBUG:teuthology.provision.fog.smithi060:FOG request: GET /image {"name": "smithi_centos_8.stream"}
2023-07-20T19:23:02.320 DEBUG:teuthology.provision.fog.smithi060:Requesting image smithi_centos_8.stream (ID 3)
2023-07-20T19:23:02.321 DEBUG:teuthology.provision.fog.smithi060:FOG request: PUT /host/165 {"imageID": 3}
2023-07-20T19:23:02.323 DEBUG:teuthology.provision.fog.smithi139:FOG request: GET /image {"name": "smithi_centos_8.stream"}
2023-07-20T19:23:02.326 DEBUG:teuthology.provision.fog.smithi119:FOG request: GET /image {"name": "smithi_centos_8.stream"}
2023-07-20T19:23:02.365 INFO:teuthology.provision.fog.smithi060:Scheduling deploy of centos 8.stream
2023-07-20T19:23:02.365 DEBUG:teuthology.provision.fog.smithi060:FOG request: GET /task/active None
2023-07-20T19:23:02.368 DEBUG:teuthology.provision.fog.smithi139:Requesting image smithi_centos_8.stream (ID 3)
2023-07-20T19:23:02.369 DEBUG:teuthology.provision.fog.smithi139:FOG request: PUT /host/244 {"imageID": 3}
2023-07-20T19:23:02.372 DEBUG:teuthology.provision.fog.smithi119:Requesting image smithi_centos_8.stream (ID 3)
2023-07-20T19:23:02.372 DEBUG:teuthology.provision.fog.smithi119:FOG request: PUT /host/224 {"imageID": 3}
2023-07-20T19:23:02.420 INFO:teuthology.provision.fog.smithi119:Scheduling deploy of centos 8.stream
2023-07-20T19:23:02.420 DEBUG:teuthology.provision.fog.smithi119:FOG request: GET /task/active None
2023-07-20T19:23:02.424 INFO:teuthology.provision.fog.smithi139:Scheduling deploy of centos 8.stream
2023-07-20T19:23:02.425 DEBUG:teuthology.provision.fog.smithi139:FOG request: GET /task/active None
2023-07-20T19:23:02.445 DEBUG:teuthology.provision.fog.smithi060:FOG request: GET /tasktype {"name": "deploy"}
2023-07-20T19:23:02.481 DEBUG:teuthology.provision.fog.smithi060:FOG request: POST /host/165/task {"taskTypeID": 1}
2023-07-20T19:23:02.486 DEBUG:teuthology.provision.fog.smithi139:FOG request: GET /tasktype {"name": "deploy"}
2023-07-20T19:23:02.490 DEBUG:teuthology.provision.fog.smithi119:FOG request: GET /tasktype {"name": "deploy"}
2023-07-20T19:23:02.522 DEBUG:teuthology.provision.fog.smithi119:FOG request: POST /host/224/task {"taskTypeID": 1}
2023-07-20T19:23:07.530 DEBUG:teuthology.provision.fog.smithi060:FOG request: GET /task/active None
2023-07-20T19:23:07.531 DEBUG:teuthology.provision.fog.smithi139:FOG request: POST /host/244/task {"taskTypeID": 1}
2023-07-20T19:23:17.602 DEBUG:teuthology.provision.fog.smithi139:FOG request: GET /task/active None
2023-07-20T19:23:17.613 DEBUG:teuthology.provision.fog.smithi119:FOG request: GET /task/active None
2023-07-20T19:23:27.654 INFO:teuthology.orchestra.console:Power off smithi060
2023-07-20T19:23:27.655 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-20T19:23:27.678 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:23:32.726 INFO:teuthology.orchestra.console:Power off smithi139
2023-07-20T19:23:32.726 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-20T19:23:32.746 INFO:teuthology.orchestra.console:Power off smithi119
2023-07-20T19:23:32.747 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-20T19:23:32.757 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:23:32.770 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:23:32.787 INFO:teuthology.orchestra.console:Power off for smithi060 completed
2023-07-20T19:23:32.880 INFO:teuthology.orchestra.console:Power off for smithi139 completed
2023-07-20T19:23:32.889 INFO:teuthology.orchestra.console:Power on smithi060
2023-07-20T19:23:32.889 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-20T19:23:32.911 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:23:32.982 INFO:teuthology.orchestra.console:Power on smithi139
2023-07-20T19:23:32.983 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-20T19:23:33.003 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:23:36.797 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:23:36.940 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:23:37.029 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:23:41.908 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:23:42.056 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:23:42.142 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:23:47.020 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:23:47.168 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:23:47.254 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:23:52.136 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:23:52.279 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:23:52.368 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:23:57.248 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:23:57.390 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:23:57.480 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:02.360 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:02.501 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:02.594 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:07.470 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:07.612 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:07.704 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:12.584 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:12.724 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:12.817 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:17.696 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:17.836 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:17.928 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:22.812 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:22.948 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:23.039 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:27.924 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:28.060 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:28.150 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:33.040 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:33.170 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:33.261 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:33.474 INFO:teuthology.orchestra.console:Power on for smithi139 completed
2023-07-20T19:24:33.575 INFO:teuthology.provision.fog.smithi139:Waiting for deploy to finish
2023-07-20T19:24:33.575 DEBUG:teuthology.provision.fog.smithi139:FOG request: GET /task/active None
2023-07-20T19:24:38.655 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-20T19:24:38.683 INFO:teuthology.orchestra.console:Power off for smithi119 completed
2023-07-20T19:24:38.784 INFO:teuthology.orchestra.console:Power on smithi119
2023-07-20T19:24:38.784 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-20T19:24:38.805 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:41.712 WARNING:teuthology.provision.fog.smithi139:[Errno None] Unable to connect to port 22 on 172.21.15.139
2023-07-20T19:24:42.584 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:42.827 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:47.698 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:47.715 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-20T19:24:52.721 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:53.206 INFO:teuthology.orchestra.console:Power on for smithi119 completed
2023-07-20T19:24:53.307 INFO:teuthology.provision.fog.smithi119:Waiting for deploy to finish
2023-07-20T19:24:53.307 DEBUG:teuthology.provision.fog.smithi119:FOG request: GET /task/active None
2023-07-20T19:24:58.248 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-07-20T19:24:58.249 DEBUG:teuthology.provision.fog.smithi060:Canceling deploy task with ID None
2023-07-20T19:24:58.249 DEBUG:teuthology.provision.fog.smithi060:FOG request: DELETE /task/cancel {"id": None}
2023-07-20T19:24:58.308 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/dispatcher/supervisor.py", line 227, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/console.py", line 243, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi060
2023-07-20T19:24:58.309 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-20T19:24:58.310 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-20T19:24:58.326 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-20T19:24:58.402 INFO:teuthology.nuke:Checking targets against current locks
2023-07-20T19:24:58.456 DEBUG:teuthology.nuke:shortname: smithi060
2023-07-20T19:24:58.456 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-20T19:24:58.492 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi060.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-20_14:36:46-fs-wip-yuri-testing-2023-07-19-1340-pacific-distro-default-smithi/7344933', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 19:22:32.235162', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='}
2023-07-20T19:24:58.493 DEBUG:teuthology.nuke:shortname: smithi119
2023-07-20T19:24:58.494 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-20T19:24:58.524 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-20_14:36:46-fs-wip-yuri-testing-2023-07-19-1340-pacific-distro-default-smithi/7344933', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 19:22:32.232207', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0='}
2023-07-20T19:24:58.525 DEBUG:teuthology.nuke:shortname: smithi139
2023-07-20T19:24:58.525 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-20T19:24:58.552 INFO:teuthology.orchestra.console:Power off smithi060
2023-07-20T19:24:58.552 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-20T19:24:58.567 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi139.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-20_14:36:46-fs-wip-yuri-testing-2023-07-19-1340-pacific-distro-default-smithi/7344933', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-20 19:22:32.233919', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'}
2023-07-20T19:24:58.597 INFO:teuthology.orchestra.console:Power off smithi119
2023-07-20T19:24:58.598 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-20T19:24:58.620 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:24:58.637 INFO:teuthology.orchestra.console:Power off smithi139
2023-07-20T19:24:58.638 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-20T19:24:58.659 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:25:01.396 WARNING:teuthology.provision.fog.smithi119:[Errno None] Unable to connect to port 22 on 172.21.15.119
2023-07-20T19:25:02.649 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:25:02.683 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:25:02.756 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:25:06.784 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:25:07.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-20T19:25:12.405 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:25:12.414 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:25:12.424 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:25:12.432 WARNING:teuthology.provision.fog.smithi139:[Errno None] Unable to connect to port 22 on 172.21.15.139
2023-07-20T19:25:12.634 INFO:teuthology.orchestra.console:Power off for smithi060 completed
2023-07-20T19:25:17.741 WARNING:teuthology.provision.fog.smithi119:[Errno None] Unable to connect to port 22 on 172.21.15.119
2023-07-20T19:25:22.769 INFO:teuthology.orchestra.console:Power off for smithi119 completed
2023-07-20T19:25:22.770 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-20T19:25:27.773 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:25:27.783 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-20T19:25:32.789 WARNING:teuthology.provision.fog.smithi119:[Errno None] Unable to connect to port 22 on 172.21.15.119
2023-07-20T19:25:32.790 WARNING:teuthology.provision.fog.smithi139:[Errno None] Unable to connect to port 22 on 172.21.15.139
2023-07-20T19:25:32.791 INFO:teuthology.lock.ops:unlocked: smithi060.front.sepia.ceph.com
2023-07-20T19:25:37.027 INFO:teuthology.lock.ops:unlocked: smithi119.front.sepia.ceph.com
2023-07-20T19:25:38.792 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-20T19:25:38.794 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-20T19:25:41.000 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:25:41.872 WARNING:teuthology.provision.fog.smithi139:[Errno None] Unable to connect to port 22 on 172.21.15.139
2023-07-20T19:25:41.876 WARNING:teuthology.provision.fog.smithi119:[Errno None] Unable to connect to port 22 on 172.21.15.119
2023-07-20T19:25:46.110 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-20T19:25:46.323 INFO:teuthology.orchestra.console:Power off for smithi139 completed
2023-07-20T19:25:46.483 INFO:teuthology.lock.ops:unlocked: smithi139.front.sepia.ceph.com
2023-07-20T19:25:46.889 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/dispatcher/supervisor.py", line 57, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/dispatcher/supervisor.py", line 227, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_79491cb69dd37a1eafb93b3471998174bdfa2fd9/teuthology/orchestra/console.py", line 243, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi060