2023-06-28T05:14:19.396 INFO:root:teuthology version: 0.0.1.dev132+gaa9345a
2023-06-28T05:14:19.398 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-06-28T05:14:19.399 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-06-28T05:14:54.570 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f20a4499670>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f20a3e02430>, signals=[15])
2023-06-28T05:14:54.571 INFO:teuthology.lock.ops:Start node 'smithi026.front.sepia.ceph.com' reimaging
2023-06-28T05:14:54.572 INFO:teuthology.lock.ops:Updating [smithi026.front.sepia.ceph.com]: reset os type and version on server
2023-06-28T05:14:54.572 INFO:teuthology.lock.ops:Updating smithi026.front.sepia.ceph.com on lock server
2023-06-28T05:14:58.267 INFO:teuthology.lock.ops:Start node 'smithi104.front.sepia.ceph.com' reimaging
2023-06-28T05:14:58.267 INFO:teuthology.lock.ops:Updating [smithi104.front.sepia.ceph.com]: reset os type and version on server
2023-06-28T05:14:58.267 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server
2023-06-28T05:14:58.269 DEBUG:teuthology.provision.fog.smithi026:FOG request: GET /host {"name": "smithi026"}
2023-06-28T05:14:58.291 INFO:teuthology.lock.ops:Start node 'smithi144.front.sepia.ceph.com' reimaging
2023-06-28T05:14:58.291 INFO:teuthology.lock.ops:Updating [smithi144.front.sepia.ceph.com]: reset os type and version on server
2023-06-28T05:14:58.292 INFO:teuthology.lock.ops:Updating smithi144.front.sepia.ceph.com on lock server
2023-06-28T05:14:58.293 DEBUG:teuthology.provision.fog.smithi104:FOG request: GET /host {"name": "smithi104"}
2023-06-28T05:14:58.316 INFO:teuthology.lock.ops:Start node 'smithi163.front.sepia.ceph.com' reimaging
2023-06-28T05:14:58.316 INFO:teuthology.lock.ops:Updating [smithi163.front.sepia.ceph.com]: reset os type and version on server
2023-06-28T05:14:58.317 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server
2023-06-28T05:14:58.318 DEBUG:teuthology.provision.fog.smithi144:FOG request: GET /host {"name": "smithi144"}
2023-06-28T05:14:58.333 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /host {"name": "smithi163"}
2023-06-28T05:14:58.346 DEBUG:teuthology.provision.fog.smithi026:FOG request: GET /image {"name": "smithi_ubuntu_20.04"}
2023-06-28T05:14:58.375 DEBUG:teuthology.provision.fog.smithi144:FOG request: GET /image {"name": "smithi_ubuntu_20.04"}
2023-06-28T05:14:58.379 DEBUG:teuthology.provision.fog.smithi104:FOG request: GET /image {"name": "smithi_ubuntu_20.04"}
2023-06-28T05:14:58.382 DEBUG:teuthology.provision.fog.smithi026:Requesting image smithi_ubuntu_20.04 (ID 2)
2023-06-28T05:14:58.382 DEBUG:teuthology.provision.fog.smithi026:FOG request: PUT /host/131 {"imageID": 2}
2023-06-28T05:14:58.397 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /image {"name": "smithi_ubuntu_20.04"}
2023-06-28T05:14:58.427 DEBUG:teuthology.provision.fog.smithi104:Requesting image smithi_ubuntu_20.04 (ID 2)
2023-06-28T05:14:58.427 DEBUG:teuthology.provision.fog.smithi104:FOG request: PUT /host/209 {"imageID": 2}
2023-06-28T05:14:58.430 DEBUG:teuthology.provision.fog.smithi144:Requesting image smithi_ubuntu_20.04 (ID 2)
2023-06-28T05:14:58.430 DEBUG:teuthology.provision.fog.smithi144:FOG request: PUT /host/249 {"imageID": 2}
2023-06-28T05:14:58.433 INFO:teuthology.provision.fog.smithi026:Scheduling deploy of ubuntu 20.04
2023-06-28T05:14:58.433 DEBUG:teuthology.provision.fog.smithi026:FOG request: GET /task/active None
2023-06-28T05:14:58.445 DEBUG:teuthology.provision.fog.smithi163:Requesting image smithi_ubuntu_20.04 (ID 2)
2023-06-28T05:14:58.445 DEBUG:teuthology.provision.fog.smithi163:FOG request: PUT /host/268 {"imageID": 2}
2023-06-28T05:14:58.483 INFO:teuthology.provision.fog.smithi144:Scheduling deploy of ubuntu 20.04
2023-06-28T05:14:58.484 DEBUG:teuthology.provision.fog.smithi144:FOG request: GET /task/active None
2023-06-28T05:14:58.486 INFO:teuthology.provision.fog.smithi104:Scheduling deploy of ubuntu 20.04
2023-06-28T05:14:58.486 DEBUG:teuthology.provision.fog.smithi104:FOG request: GET /task/active None
2023-06-28T05:14:58.494 INFO:teuthology.provision.fog.smithi163:Scheduling deploy of ubuntu 20.04
2023-06-28T05:14:58.494 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /task/active None
2023-06-28T05:14:58.522 DEBUG:teuthology.provision.fog.smithi026:FOG request: GET /tasktype {"name": "deploy"}
2023-06-28T05:14:58.569 DEBUG:teuthology.provision.fog.smithi026:FOG request: POST /host/131/task {"taskTypeID": 1}
2023-06-28T05:14:58.577 DEBUG:teuthology.provision.fog.smithi144:FOG request: GET /tasktype {"name": "deploy"}
2023-06-28T05:14:58.581 DEBUG:teuthology.provision.fog.smithi104:FOG request: GET /tasktype {"name": "deploy"}
2023-06-28T05:14:58.588 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /tasktype {"name": "deploy"}
2023-06-28T05:14:58.619 DEBUG:teuthology.provision.fog.smithi104:FOG request: POST /host/209/task {"taskTypeID": 1}
2023-06-28T05:14:58.622 DEBUG:teuthology.provision.fog.smithi163:FOG request: POST /host/268/task {"taskTypeID": 1}
2023-06-28T05:14:58.626 DEBUG:teuthology.provision.fog.smithi144:FOG request: POST /host/249/task {"taskTypeID": 1}
2023-06-28T05:14:58.628 DEBUG:teuthology.provision.fog.smithi026:FOG request: GET /task/active None
2023-06-28T05:14:58.765 DEBUG:teuthology.provision.fog.smithi144:FOG request: GET /task/active None
2023-06-28T05:14:58.768 DEBUG:teuthology.provision.fog.smithi104:FOG request: GET /task/active None
2023-06-28T05:14:58.769 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /task/active None
2023-06-28T05:14:58.843 INFO:teuthology.orchestra.console:Power off smithi026
2023-06-28T05:14:58.844 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-06-28T05:14:58.867 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:14:58.922 INFO:teuthology.orchestra.console:Power off smithi144
2023-06-28T05:14:58.923 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-06-28T05:14:58.942 INFO:teuthology.orchestra.console:Power off smithi104
2023-06-28T05:14:58.943 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-06-28T05:14:58.957 INFO:teuthology.orchestra.console:Power off smithi163
2023-06-28T05:14:58.957 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-06-28T05:14:58.966 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:14:58.975 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:14:58.984 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:14:58.993 INFO:teuthology.orchestra.console:Power off for smithi026 completed
2023-06-28T05:14:59.093 INFO:teuthology.orchestra.console:Power off for smithi104 completed
2023-06-28T05:14:59.094 INFO:teuthology.orchestra.console:Power on smithi026
2023-06-28T05:14:59.094 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-06-28T05:14:59.104 INFO:teuthology.orchestra.console:Power off for smithi144 completed
2023-06-28T05:14:59.105 INFO:teuthology.orchestra.console:Power off for smithi163 completed
2023-06-28T05:14:59.117 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:14:59.194 INFO:teuthology.orchestra.console:Power on smithi104
2023-06-28T05:14:59.195 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-06-28T05:14:59.205 INFO:teuthology.orchestra.console:Power on smithi144
2023-06-28T05:14:59.206 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-06-28T05:14:59.217 INFO:teuthology.orchestra.console:Power on smithi163
2023-06-28T05:14:59.218 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-06-28T05:14:59.227 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:14:59.237 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:14:59.248 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:15:03.143 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:15:03.259 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:15:03.269 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:15:03.278 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:15:08.253 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:15:08.370 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:15:08.379 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:15:08.388 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:15:08.583 INFO:teuthology.orchestra.console:Power on for smithi104 completed
2023-06-28T05:15:08.684 INFO:teuthology.provision.fog.smithi104:Waiting for deploy to finish
2023-06-28T05:15:08.684 DEBUG:teuthology.provision.fog.smithi104:FOG request: GET /task/active None
2023-06-28T05:15:13.365 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:15:13.493 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:15:13.501 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:15:13.704 INFO:teuthology.orchestra.console:Power on for smithi144 completed
2023-06-28T05:15:13.805 INFO:teuthology.provision.fog.smithi144:Waiting for deploy to finish
2023-06-28T05:15:13.806 DEBUG:teuthology.provision.fog.smithi144:FOG request: GET /task/active None
2023-06-28T05:15:18.914 INFO:teuthology.orchestra.console:Power on for smithi163 completed
2023-06-28T05:15:19.015 INFO:teuthology.provision.fog.smithi163:Waiting for deploy to finish
2023-06-28T05:15:19.015 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /task/active None
2023-06-28T05:15:24.023 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:15:24.031 DEBUG:teuthology.provision.fog.smithi104:FOG request: GET /task/active None
2023-06-28T05:15:33.916 DEBUG:teuthology.provision.fog.smithi144:FOG request: GET /task/active None
2023-06-28T05:15:38.921 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:15:44.036 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:15:44.161 DEBUG:teuthology.provision.fog.smithi104:FOG request: GET /task/active None
2023-06-28T05:15:48.998 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /task/active None
2023-06-28T05:15:53.004 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:15:54.033 DEBUG:teuthology.provision.fog.smithi144:FOG request: GET /task/active None
2023-06-28T05:16:03.044 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:16:04.201 DEBUG:teuthology.provision.fog.smithi104:FOG request: GET /task/active None
2023-06-28T05:16:09.204 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /task/active None
2023-06-28T05:16:09.206 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:16:14.217 DEBUG:teuthology.provision.fog.smithi144:FOG request: GET /task/active None
2023-06-28T05:16:23.230 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:16:28.344 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:16:33.455 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:16:34.230 DEBUG:teuthology.provision.fog.smithi104:FOG request: GET /task/active None
2023-06-28T05:16:35.194 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /task/active None
2023-06-28T05:16:35.196 DEBUG:teuthology.provision.fog.smithi144:FOG request: GET /task/active None
2023-06-28T05:16:39.200 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:16:44.312 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:16:45.424 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-06-28T05:16:45.425 DEBUG:teuthology.provision.fog.smithi026:Canceling deploy task with ID 349645
2023-06-28T05:16:45.426 DEBUG:teuthology.provision.fog.smithi026:FOG request: DELETE /task/cancel {"id": 349645}
2023-06-28T05:16:49.529 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 227, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/console.py", line 243, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi026
2023-06-28T05:16:49.530 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-06-28T05:16:49.531 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-06-28T05:16:49.616 INFO:teuthology.nuke:Checking targets against current locks
2023-06-28T05:16:59.678 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /task/active None
2023-06-28T05:16:59.680 DEBUG:teuthology.provision.fog.smithi144:FOG request: GET /task/active None
2023-06-28T05:16:59.681 DEBUG:teuthology.provision.fog.smithi104:FOG request: GET /task/active None
2023-06-28T05:17:02.211 DEBUG:teuthology.nuke:shortname: smithi026
2023-06-28T05:17:02.211 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-06-28T05:17:05.508 DEBUG:teuthology.nuke:shortname: smithi104
2023-06-28T05:17:05.509 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-06-28T05:17:05.531 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi026.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-27_22:53:08-rados-reef-release-distro-default-smithi/7318909', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 05:14:18.522550', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJb/iRFuJLfi1KVMydpldGSHRe6O/pkhpjvRacxfZL1hkgFU0DLFtxZ1/Yle/7EMTYkEdMFu2DDVdESNmopAP2g='}
2023-06-28T05:17:05.535 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-27_22:53:08-rados-reef-release-distro-default-smithi/7318909', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 05:14:18.524515', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}
2023-06-28T05:17:05.536 DEBUG:teuthology.nuke:shortname: smithi144
2023-06-28T05:17:05.536 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-06-28T05:17:05.565 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi144.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-27_22:53:08-rados-reef-release-distro-default-smithi/7318909', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 05:14:18.525749', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'}
2023-06-28T05:17:05.566 DEBUG:teuthology.nuke:shortname: smithi163
2023-06-28T05:17:05.567 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-06-28T05:17:05.596 INFO:teuthology.orchestra.console:Power off smithi104
2023-06-28T05:17:05.596 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-06-28T05:17:05.608 INFO:teuthology.orchestra.console:Power off smithi026
2023-06-28T05:17:05.608 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-06-28T05:17:05.615 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-27_22:53:08-rados-reef-release-distro-default-smithi/7318909', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 05:14:18.520167', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBINPX12r4ZVhYJE1g0nifCUgHyr+NkE1Bv8i7kaMVJT7sAga74c0/DgoRVMoR45iMlnpLEj+deSGUHLc9cADyQY='}
2023-06-28T05:17:05.619 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:17:05.629 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:17:05.649 INFO:teuthology.orchestra.console:Power off smithi144
2023-06-28T05:17:05.650 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-06-28T05:17:05.673 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:17:05.690 INFO:teuthology.orchestra.console:Power off smithi163
2023-06-28T05:17:05.690 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-06-28T05:17:05.711 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:17:09.644 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:17:09.664 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:17:09.700 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:17:09.732 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:17:13.756 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:17:13.774 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:17:13.810 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:17:13.842 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:17:18.868 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:17:18.886 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:17:18.920 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:17:18.952 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:17:19.078 INFO:teuthology.orchestra.console:Power off for smithi104 completed
2023-06-28T05:17:24.185 DEBUG:teuthology.provision.fog.smithi104:FOG request: GET /task/active None
2023-06-28T05:17:24.187 DEBUG:teuthology.provision.fog.smithi144:FOG request: GET /task/active None
2023-06-28T05:17:24.187 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /task/active None
2023-06-28T05:17:28.272 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:17:28.282 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:17:28.292 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:17:31.032 INFO:teuthology.lock.ops:unlocked: smithi104.front.sepia.ceph.com
2023-06-28T05:17:31.106 INFO:teuthology.orchestra.console:Power off for smithi144 completed
2023-06-28T05:17:31.257 INFO:teuthology.lock.ops:unlocked: smithi144.front.sepia.ceph.com
2023-06-28T05:17:36.004 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:17:36.011 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:17:41.114 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:17:41.124 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:17:46.006 DEBUG:teuthology.provision.fog.smithi144:FOG request: GET /task/active None
2023-06-28T05:17:46.008 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /task/active None
2023-06-28T05:17:46.009 DEBUG:teuthology.provision.fog.smithi104:FOG request: GET /task/active None
2023-06-28T05:18:01.022 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:18:01.030 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:18:06.132 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:18:06.140 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:18:11.244 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:18:11.253 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:18:16.124 DEBUG:teuthology.provision.fog.smithi144:FOG request: GET /task/active None
2023-06-28T05:18:21.133 DEBUG:teuthology.provision.fog.smithi104:FOG request: GET /task/active None
2023-06-28T05:18:21.134 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /task/active None
2023-06-28T05:18:21.135 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:18:21.143 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:18:21.356 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-06-28T05:18:30.364 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:18:30.372 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:18:35.475 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:18:35.483 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:18:36.354 DEBUG:teuthology.provision.fog.smithi144:FOG request: GET /task/active None
2023-06-28T05:18:41.364 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /task/active None
2023-06-28T05:18:46.370 INFO:teuthology.orchestra.console:Power off for smithi163 completed
2023-06-28T05:18:46.370 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:18:46.378 WARNING:teuthology.provision.fog.smithi104:[Errno None] Unable to connect to port 22 on 172.21.15.104
2023-06-28T05:18:55.573 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-06-28T05:18:55.603 INFO:teuthology.lock.ops:unlocked: smithi163.front.sepia.ceph.com
2023-06-28T05:18:58.640 WARNING:teuthology.provision.fog.smithi104:[Errno None] Unable to connect to port 22 on 172.21.15.104
2023-06-28T05:18:59.577 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-28T05:19:00.687 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-06-28T05:19:00.688 ERROR:teuthology.orchestra.console:Failed to power off smithi026
2023-06-28T05:19:05.693 DEBUG:teuthology.provision.fog.smithi144:FOG request: GET /task/active None
2023-06-28T05:19:05.695 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /task/active None
2023-06-28T05:19:05.697 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-06-28T05:19:17.444 WARNING:teuthology.provision.fog.smithi104:[Errno None] Unable to connect to port 22 on 172.21.15.104
2023-06-28T05:19:22.486 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-06-28T05:19:27.491 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-06-28T05:19:32.523 INFO:teuthology.lock.ops:unlocked: smithi026.front.sepia.ceph.com
2023-06-28T05:19:37.485 DEBUG:teuthology.provision.fog.smithi163:FOG request: GET /task/active None
2023-06-28T05:19:48.689 WARNING:teuthology.provision.fog.smithi144:[Errno None] Unable to connect to port 22 on 172.21.15.144
2023-06-28T05:19:50.845 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 57, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 227, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/console.py", line 243, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi026