2023-07-09T10:38:40.694 INFO:root:teuthology version: 0.0.1.dev140+gb53f236
2023-07-09T10:38:40.696 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-09T10:38:40.696 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-09T10:38:40.893 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f721ba0c5e0>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f721b36d670>, signals=[15])
2023-07-09T10:38:40.894 INFO:teuthology.lock.ops:Start node 'smithi016.front.sepia.ceph.com' reimaging
2023-07-09T10:38:40.895 INFO:teuthology.lock.ops:Updating [smithi016.front.sepia.ceph.com]: reset os type and version on server
2023-07-09T10:38:40.895 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server
2023-07-09T10:38:40.923 INFO:teuthology.lock.ops:Start node 'smithi067.front.sepia.ceph.com' reimaging
2023-07-09T10:38:40.923 INFO:teuthology.lock.ops:Updating [smithi067.front.sepia.ceph.com]: reset os type and version on server
2023-07-09T10:38:40.923 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server
2023-07-09T10:38:40.925 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /host {"name": "smithi016"}
2023-07-09T10:38:40.948 INFO:teuthology.lock.ops:Start node 'smithi123.front.sepia.ceph.com' reimaging
2023-07-09T10:38:40.948 INFO:teuthology.lock.ops:Updating [smithi123.front.sepia.ceph.com]: reset os type and version on server
2023-07-09T10:38:40.948 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server
2023-07-09T10:38:40.949 DEBUG:teuthology.provision.fog.smithi067:FOG request: GET /host {"name": "smithi067"}
2023-07-09T10:38:40.973 DEBUG:teuthology.provision.fog.smithi123:FOG request: GET /host {"name": "smithi123"}
2023-07-09T10:38:41.004 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /image {"name": "smithi_ubuntu_20.04"}
2023-07-09T10:38:41.018 DEBUG:teuthology.provision.fog.smithi067:FOG request: GET /image {"name": "smithi_ubuntu_20.04"}
2023-07-09T10:38:41.046 DEBUG:teuthology.provision.fog.smithi123:FOG request: GET /image {"name": "smithi_ubuntu_20.04"}
2023-07-09T10:38:41.048 DEBUG:teuthology.provision.fog.smithi016:Requesting image smithi_ubuntu_20.04 (ID 2)
2023-07-09T10:38:41.048 DEBUG:teuthology.provision.fog.smithi016:FOG request: PUT /host/122 {"imageID": 2}
2023-07-09T10:38:41.055 DEBUG:teuthology.provision.fog.smithi067:Requesting image smithi_ubuntu_20.04 (ID 2)
2023-07-09T10:38:41.055 DEBUG:teuthology.provision.fog.smithi067:FOG request: PUT /host/172 {"imageID": 2}
2023-07-09T10:38:41.090 INFO:teuthology.provision.fog.smithi016:Scheduling deploy of ubuntu 20.04
2023-07-09T10:38:41.090 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /task/active None
2023-07-09T10:38:41.095 DEBUG:teuthology.provision.fog.smithi123:Requesting image smithi_ubuntu_20.04 (ID 2)
2023-07-09T10:38:41.095 DEBUG:teuthology.provision.fog.smithi123:FOG request: PUT /host/228 {"imageID": 2}
2023-07-09T10:38:41.107 INFO:teuthology.provision.fog.smithi067:Scheduling deploy of ubuntu 20.04
2023-07-09T10:38:41.108 DEBUG:teuthology.provision.fog.smithi067:FOG request: GET /task/active None
2023-07-09T10:38:41.133 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /tasktype {"name": "deploy"}
2023-07-09T10:38:41.140 INFO:teuthology.provision.fog.smithi123:Scheduling deploy of ubuntu 20.04
2023-07-09T10:38:41.140 DEBUG:teuthology.provision.fog.smithi123:FOG request: GET /task/active None
2023-07-09T10:38:41.144 DEBUG:teuthology.provision.fog.smithi067:FOG request: GET /tasktype {"name": "deploy"}
2023-07-09T10:38:41.168 DEBUG:teuthology.provision.fog.smithi016:FOG request: POST /host/122/task {"taskTypeID": 1}
2023-07-09T10:38:41.175 DEBUG:teuthology.provision.fog.smithi123:FOG request: GET /tasktype {"name": "deploy"}
2023-07-09T10:38:41.178 DEBUG:teuthology.provision.fog.smithi067:FOG request: POST /host/172/task {"taskTypeID": 1}
2023-07-09T10:38:41.210 DEBUG:teuthology.provision.fog.smithi123:FOG request: POST /host/228/task {"taskTypeID": 1}
2023-07-09T10:38:41.243 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /task/active None
2023-07-09T10:38:41.254 DEBUG:teuthology.provision.fog.smithi067:FOG request: GET /task/active None
2023-07-09T10:38:41.266 DEBUG:teuthology.provision.fog.smithi123:FOG request: GET /task/active None
2023-07-09T10:38:41.327 INFO:teuthology.orchestra.console:Power off smithi016
2023-07-09T10:38:41.328 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-09T10:38:41.344 INFO:teuthology.orchestra.console:Power off smithi067
2023-07-09T10:38:41.344 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-09T10:38:41.359 INFO:teuthology.orchestra.console:Power off smithi123
2023-07-09T10:38:41.359 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-09T10:38:41.368 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:38:41.376 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:38:41.386 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:38:45.400 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:38:45.408 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:38:45.415 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:38:50.511 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:38:50.520 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:38:50.529 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:38:55.624 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:38:55.633 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:38:55.643 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:00.740 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:00.749 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:00.756 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:05.852 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:05.861 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:05.870 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:06.080 INFO:teuthology.orchestra.console:Power off for smithi123 completed
2023-07-09T10:39:06.181 INFO:teuthology.orchestra.console:Power on smithi123
2023-07-09T10:39:06.181 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-09T10:39:06.201 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:10.224 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:10.965 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:10.973 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:15.336 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:16.076 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:16.086 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:20.448 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:21.190 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:21.200 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:25.560 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:26.303 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:26.313 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:26.525 INFO:teuthology.orchestra.console:Power off for smithi016 completed
2023-07-09T10:39:26.626 INFO:teuthology.orchestra.console:Power on smithi016
2023-07-09T10:39:26.626 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-09T10:39:26.646 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:30.672 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:30.680 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:31.415 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:35.783 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:35.790 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:36.528 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:40.894 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:40.904 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:41.639 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:46.007 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:46.016 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:46.750 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:51.119 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:51.130 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:51.863 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:52.976 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-07-09T10:39:53.009 ERROR:teuthology.orchestra.console:Failed to power off smithi067
2023-07-09T10:39:53.009 INFO:teuthology.orchestra.console:Power on smithi067
2023-07-09T10:39:53.009 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-09T10:39:56.233 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:56.243 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:39:57.007 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:40:01.032 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:40:01.346 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:40:01.354 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:40:06.144 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:40:06.456 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:40:06.466 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:40:11.256 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:40:11.568 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:40:11.577 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:40:11.780 INFO:teuthology.orchestra.console:Power on for smithi016 completed
2023-07-09T10:40:11.881 INFO:teuthology.provision.fog.smithi016:Waiting for deploy to finish
2023-07-09T10:40:11.882 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /task/active None
2023-07-09T10:40:20.188 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:40:20.197 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:40:21.307 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-07-09T10:40:21.308 DEBUG:teuthology.provision.fog.smithi123:Canceling deploy task with ID 371027
2023-07-09T10:40:21.308 DEBUG:teuthology.provision.fog.smithi123:FOG request: DELETE /task/cancel {"id": 371027}
2023-07-09T10:40:36.319 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:40:36.327 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /task/active None
2023-07-09T10:40:41.384 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/dispatcher/supervisor.py", line 227, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/console.py", line 243, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi123
2023-07-09T10:40:41.385 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-09T10:40:41.386 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-09T10:40:46.254 INFO:teuthology.orchestra.console:Power on for smithi067 completed
2023-07-09T10:40:46.325 INFO:teuthology.nuke:Checking targets against current locks
2023-07-09T10:40:51.333 INFO:teuthology.provision.fog.smithi067:Waiting for deploy to finish
2023-07-09T10:40:51.334 DEBUG:teuthology.provision.fog.smithi067:FOG request: GET /task/active None
2023-07-09T10:41:06.374 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /task/active None
2023-07-09T10:41:16.419 DEBUG:teuthology.nuke:shortname: smithi016
2023-07-09T10:41:16.419 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-09T10:41:17.529 DEBUG:teuthology.nuke:shortname: smithi067
2023-07-09T10:41:17.530 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-09T10:41:17.534 DEBUG:teuthology.nuke:shortname: smithi123
2023-07-09T10:41:17.535 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-09T10:41:17.541 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-08_17:44:31-smoke-reef-distro-default-smithi/7330254', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-09 10:38:29.777099', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD6ugUoyH4ocA5DDZwZtXXWJPNlKqsZTKXpz89wBFJQXm/GAi2Y01SYUh6Hwmz8TSyqG++DYVRMaD4QvG7qkfA1D1rfMtmWcnd0I/LV3uYnFs0D1HvdO/nnIHcwEpjOA8F+G37lGrzq1XLvpRduOYCIQ9ONd3UbB+OCPFumVpQgnzjZFFIeNn3sEnlTuLzE3UkRmCpqsS0J6alvuKybl1ZgSuateujlFl2IO8FksTQxwgMcgO5dSObCSQZcVlAAPDE2FLSpB3+hUIFBrdz8od+lB82mLltBt2XC94IF3qfAyg5QrPhQjv2NAW+rnbaF6vCRK4QN/38xNTcYRp/9UNxILqdmOEMQ9WmJCYgtyoCiRJfe70vJ6VDY6RVnerJBzPJ6LAoWYyAB1rnwcDUAflTh69wuCWkTXswfkc0zRrd8aLsK4GOHnphl73e4JDJMSCh+KZgdObgFeiPQuJ6qhmMyBdmuzv7UinrEfu2fVOA0Pr2tnds9jzKPOAZRO4y+iAk='}
2023-07-09T10:41:17.554 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-08_17:44:31-smoke-reef-distro-default-smithi/7330254', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-09 10:38:29.778472', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTQn3yWPjcSBOylCvii6l4gXdVqJP1EG3hzBkwbrc27MXuDzr8gS7qz8nAlqn6bCl1zwU6qWSJOqxqs7Y8ZqlvX5mjnMuScyP6Y9A5P8JpGJe3P/J+CXHpK02TviMPAikZ/uqvMJXV9b5+1SZ7m/BGrF94xjE7ot3i1huQPOCiLqjd/1nW6+/m/prBe9U3ShwybNnby3GX6XZJdYnPf2lMj+s2tDZGR4pg7ccDq4rUnNNCF7jVCiTi9HEpQiLLLADAK1nFAS9YbwFAm96TE9Jq2CISyYbldgHTt+SAnEDyswaqlHkLctPNBOsFkpgxb0gzdLDpjs06fcTPYjljzu8jRC61t2cCynbupRjpeBYAGEmr2QZpGEIs8aK4U1sVGeqJ7HSY/TVIOzEPhHvCzc2OCN+TzRK4ntC7zEQZ/4OFmQ3mB+gVm1+8hmhylM+dlgZ/BnuqjK41/yJbK0U1wegHZ9oBkk//BDGUmfoReZQJxG4rzAw8sA6IQo2faglOGnk='}
2023-07-09T10:41:17.558 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-08_17:44:31-smoke-reef-distro-default-smithi/7330254', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-09 10:38:29.775317', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNGMebNWV9QGoDmUAlxcTCW3T1pbsQTMUX2S1uIzo09+pcUQ6Vba+wU6c5HZLM/VoO0Ki7aEoU3u/N9oZduRQt+Fd2Mq0mNm8g1QCRQpqn1HEBKoAoS7tebb77MZ4//78rmjB40Ok8YHOzmAoCpbqk3zk1I+6mKsxgzWBU/Gvy1v4yu1oUhqFmTWOjgRM2oGM6MtsYLgiZDsTR6zllZZvxZ0H7pIRRzYbnDINBedj2N+T17/MsvJaDgR/pBV0B7IwPhHjpBWPgLUOmRFIFrfPJknuvnvQX/M9/TVRS03dcC9/J5d0k6MscXTrNFdH70y9cqPZ8shlre5/NwcBbb05oQJLl7LpGGhqH3jf4B5efYFvupIGwxMo4bpHlI6P89ngVp/Z3ES2hsxCm6Yfyr30mcggcuAf6BivHmpzo+nDhuDgmK2NV3HHpB5+SFkF2Tno8XndqeVbbY9HIZbQQY9N5C/uCq1ARLbAxoUYkdmiAD8+4V2p9sT22T3WE8b1qyYs='}
2023-07-09T10:41:17.590 INFO:teuthology.orchestra.console:Power off smithi016
2023-07-09T10:41:17.591 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-09T10:41:17.613 INFO:teuthology.orchestra.console:Power off smithi067
2023-07-09T10:41:17.613 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-09T10:41:17.625 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:41:17.638 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:41:17.661 INFO:teuthology.orchestra.console:Power off smithi123
2023-07-09T10:41:17.661 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-09T10:41:17.684 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:41:21.384 DEBUG:teuthology.provision.fog.smithi067:FOG request: GET /task/active None
2023-07-09T10:41:21.673 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:41:21.682 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:41:22.695 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:41:25.787 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:41:25.796 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:41:26.807 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:41:27.019 INFO:teuthology.orchestra.console:Power off for smithi123 completed
2023-07-09T10:41:28.558 INFO:teuthology.lock.ops:unlocked: smithi123.front.sepia.ceph.com
2023-07-09T10:41:30.899 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:41:30.908 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:41:31.419 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /task/active None
2023-07-09T10:41:36.446 DEBUG:teuthology.provision.fog.smithi067:FOG request: GET /task/active None
2023-07-09T10:41:41.453 INFO:teuthology.orchestra.console:Power off for smithi016 completed
2023-07-09T10:41:41.453 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-09T10:41:51.621 INFO:teuthology.lock.ops:unlocked: smithi016.front.sepia.ceph.com
2023-07-09T10:41:51.689 INFO:teuthology.orchestra.console:Power off for smithi067 completed
2023-07-09T10:41:56.796 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /task/active None
2023-07-09T10:41:56.797 DEBUG:teuthology.provision.fog.smithi067:FOG request: GET /task/active None
2023-07-09T10:42:11.869 INFO:teuthology.lock.ops:unlocked: smithi067.front.sepia.ceph.com
2023-07-09T10:42:26.844 DEBUG:teuthology.provision.fog.smithi016:FOG request: GET /task/active None
2023-07-09T10:42:26.846 DEBUG:teuthology.provision.fog.smithi067:FOG request: GET /task/active None
2023-07-09T10:42:31.913 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/dispatcher/supervisor.py", line 57, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/dispatcher/supervisor.py", line 227, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/console.py", line 243, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi123