2023-09-21T13:18:00.362 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-21T13:18:00.364 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-09-21T13:18:00.504 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7fb55bcc8cd0>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7fb55c7cd670>, signals=[15])
2023-09-21T13:18:00.506 INFO:teuthology.lock.ops:Start node 'smithi045.front.sepia.ceph.com' reimaging
2023-09-21T13:18:00.506 INFO:teuthology.lock.ops:Updating [smithi045.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T13:18:00.506 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server
2023-09-21T13:18:00.532 INFO:teuthology.lock.ops:Start node 'smithi110.front.sepia.ceph.com' reimaging
2023-09-21T13:18:00.533 INFO:teuthology.lock.ops:Updating [smithi110.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T13:18:00.533 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server
2023-09-21T13:18:00.536 DEBUG:teuthology.provision.fog.smithi045:FOG request: GET /host {"name": "smithi045"}
2023-09-21T13:18:00.558 DEBUG:teuthology.provision.fog.smithi110:FOG request: GET /host {"name": "smithi110"}
2023-09-21T13:18:00.611 DEBUG:teuthology.provision.fog.smithi045:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-09-21T13:18:00.629 DEBUG:teuthology.provision.fog.smithi110:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-09-21T13:18:00.652 DEBUG:teuthology.provision.fog.smithi045:Requesting image smithi_rhel_8.6 (ID 5)
2023-09-21T13:18:00.652 DEBUG:teuthology.provision.fog.smithi045:FOG request: PUT /host/150 {"imageID": 5}
2023-09-21T13:18:00.673 DEBUG:teuthology.provision.fog.smithi110:Requesting image smithi_rhel_8.6 (ID 5)
2023-09-21T13:18:00.674 DEBUG:teuthology.provision.fog.smithi110:FOG request: PUT /host/215 {"imageID": 5}
2023-09-21T13:18:00.690 INFO:teuthology.provision.fog.smithi045:Scheduling deploy of rhel 8.6
2023-09-21T13:18:00.690 DEBUG:teuthology.provision.fog.smithi045:FOG request: GET /task/active None
2023-09-21T13:18:00.724 INFO:teuthology.provision.fog.smithi110:Scheduling deploy of rhel 8.6
2023-09-21T13:18:00.725 DEBUG:teuthology.provision.fog.smithi110:FOG request: GET /task/active None
2023-09-21T13:18:00.758 DEBUG:teuthology.provision.fog.smithi045:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T13:18:00.790 DEBUG:teuthology.provision.fog.smithi110:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T13:18:00.802 DEBUG:teuthology.provision.fog.smithi045:FOG request: POST /host/150/task {"taskTypeID": 1}
2023-09-21T13:18:00.831 DEBUG:teuthology.provision.fog.smithi110:FOG request: POST /host/215/task {"taskTypeID": 1}
2023-09-21T13:18:00.847 DEBUG:teuthology.provision.fog.smithi045:FOG request: GET /task/active None
2023-09-21T13:18:00.882 DEBUG:teuthology.provision.fog.smithi110:FOG request: GET /task/active None
2023-09-21T13:18:00.944 INFO:teuthology.orchestra.console.smithi045:Power off
2023-09-21T13:18:00.945 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T13:18:00.969 DEBUG:teuthology.orchestra.console.smithi045:power off output: Chassis Power Control: Down/Off
2023-09-21T13:18:00.970 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:18:00.991 INFO:teuthology.orchestra.console.smithi110:Power off
2023-09-21T13:18:00.991 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T13:18:01.000 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off
2023-09-21T13:18:01.101 INFO:teuthology.orchestra.console.smithi045:Power off completed
2023-09-21T13:18:01.202 INFO:teuthology.orchestra.console.smithi045:Power on
2023-09-21T13:18:01.203 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T13:18:01.240 DEBUG:teuthology.orchestra.console.smithi045:power on output: Chassis Power Control: Up/On
2023-09-21T13:18:01.241 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:18:01.263 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off
2023-09-21T13:18:05.266 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:18:05.375 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on
2023-09-21T13:18:05.476 INFO:teuthology.orchestra.console.smithi045:Power on completed
2023-09-21T13:18:05.577 INFO:teuthology.provision.fog.smithi045:Waiting for deploy to finish
2023-09-21T13:18:05.577 DEBUG:teuthology.provision.fog.smithi045:FOG request: GET /task/active None
2023-09-21T13:18:20.658 DEBUG:teuthology.provision.fog.smithi045:FOG request: GET /task/active None
2023-09-21T13:18:21.064 DEBUG:teuthology.orchestra.console.smithi110:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T13:18:21.064 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T13:18:35.760 DEBUG:teuthology.provision.fog.smithi045:FOG request: GET /task/active None
2023-09-21T13:18:41.136 DEBUG:teuthology.orchestra.console.smithi110:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T13:18:41.136 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T13:18:50.874 DEBUG:teuthology.provision.fog.smithi045:FOG request: GET /task/active None
2023-09-21T13:19:01.209 DEBUG:teuthology.orchestra.console.smithi110:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T13:19:01.210 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:19:02.218 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:19:05.954 DEBUG:teuthology.provision.fog.smithi045:FOG request: GET /task/active None
2023-09-21T13:19:06.219 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:19:07.328 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:19:11.330 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:19:12.440 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:19:16.442 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:19:17.550 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:19:21.025 DEBUG:teuthology.provision.fog.smithi045:FOG request: GET /task/active None
2023-09-21T13:19:21.552 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:19:22.661 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:19:26.666 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:19:27.776 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:19:31.779 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:19:32.888 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:19:36.094 DEBUG:teuthology.provision.fog.smithi045:FOG request: GET /task/active None
2023-09-21T13:19:36.890 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:19:38.000 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:19:42.002 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:19:43.112 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:19:47.114 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:19:48.224 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:19:51.168 DEBUG:teuthology.provision.fog.smithi045:FOG request: GET /task/active None
2023-09-21T13:19:52.226 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:19:53.334 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:19:57.339 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:19:58.449 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:20:02.450 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:20:03.561 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:20:06.250 DEBUG:teuthology.provision.fog.smithi045:FOG request: GET /task/active None
2023-09-21T13:20:07.563 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:20:08.674 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:20:12.678 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:20:13.790 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:20:13.791 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T13:20:13.791 ERROR:teuthology.orchestra.console.smithi110:Failed to power off
2023-09-21T13:20:13.792 INFO:teuthology.orchestra.console.smithi110:Power on
2023-09-21T13:20:13.792 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T13:20:21.346 DEBUG:teuthology.provision.fog.smithi045:FOG request: GET /task/active None
2023-09-21T13:20:33.865 DEBUG:teuthology.orchestra.console.smithi110:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T13:20:33.866 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T13:20:36.450 DEBUG:teuthology.provision.fog.smithi045:FOG request: GET /task/active None
2023-09-21T13:20:36.545 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T13:20:53.936 DEBUG:teuthology.orchestra.console.smithi110:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T13:20:53.954 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T13:20:54.938 WARNING:teuthology.provision.fog.smithi045:[Errno None] Unable to connect to port 22 on 172.21.15.45
2023-09-21T13:21:00.943 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T13:21:03.994 WARNING:teuthology.provision.fog.smithi045:[Errno None] Unable to connect to port 22 on 172.21.15.45
2023-09-21T13:21:09.998 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T13:21:13.050 WARNING:teuthology.provision.fog.smithi045:[Errno None] Unable to connect to port 22 on 172.21.15.45
2023-09-21T13:21:14.028 DEBUG:teuthology.orchestra.console.smithi110:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T13:21:14.028 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:21:15.037 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:21:19.042 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:21:19.052 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T13:21:20.153 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:21:22.106 WARNING:teuthology.provision.fog.smithi045:[Errno None] Unable to connect to port 22 on 172.21.15.45
2023-09-21T13:21:24.157 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:21:25.267 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:21:28.111 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T13:21:29.269 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:21:30.379 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:21:34.382 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:21:35.491 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:21:39.497 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:21:40.606 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:21:44.610 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:21:45.720 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:21:49.722 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:21:50.831 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:21:54.834 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:21:55.946 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:21:59.950 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:22:01.059 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:22:05.062 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:22:06.172 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:22:10.174 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:22:11.283 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:22:15.286 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:22:16.394 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:22:20.398 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:22:21.507 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:22:25.511 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:22:26.621 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:22:26.622 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T13:22:26.622 DEBUG:teuthology.provision.fog.smithi110:Canceling deploy task with ID 499722
2023-09-21T13:22:26.623 DEBUG:teuthology.provision.fog.smithi110:FOG request: DELETE /task/cancel {"id": 499722}
2023-09-21T13:22:26.680 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi110
2023-09-21T13:22:26.681 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-09-21T13:22:26.831 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=9db37401dd76419bad6ff3b62f19c3de
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi110
2023-09-21T13:22:26.834 INFO:teuthology.nuke:Checking targets against current locks
2023-09-21T13:22:26.892 DEBUG:teuthology.nuke:shortname: smithi045
2023-09-21T13:22:26.893 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T13:22:26.915 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/svelar-2023-09-20_07:59:56-rados-wip-svelar-testing-1-2023-09-20-1125-distro-default-smithi/7398643', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 13:17:59.475791', 'locked_by': 'scheduled_svelar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}
2023-09-21T13:22:26.923 DEBUG:teuthology.nuke:shortname: smithi110
2023-09-21T13:22:26.923 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T13:22:26.947 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/svelar-2023-09-20_07:59:56-rados-wip-svelar-testing-1-2023-09-20-1125-distro-default-smithi/7398643', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 13:17:59.477775', 'locked_by': 'scheduled_svelar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLH15lhs6mMmDHFGpdBB16wtVR7ydtDsuyvKFjdKxRrfl7qrh+qcKGU5yB0SJo7JVYjbBB/bEzhiFCKoL0P0Fzk='}
2023-09-21T13:22:26.962 INFO:teuthology.orchestra.console.smithi045:Power off
2023-09-21T13:22:26.963 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T13:22:26.986 DEBUG:teuthology.orchestra.console.smithi045:power off output: Chassis Power Control: Down/Off
2023-09-21T13:22:26.987 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:22:27.006 INFO:teuthology.orchestra.console.smithi110:Power off
2023-09-21T13:22:27.006 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T13:22:27.016 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on
2023-09-21T13:22:28.115 WARNING:teuthology.provision.fog.smithi045:timed out
2023-09-21T13:22:31.018 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:22:31.128 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on
2023-09-21T13:22:34.131 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T13:22:35.130 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:22:35.239 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off
2023-09-21T13:22:35.340 INFO:teuthology.orchestra.console.smithi045:Power off completed
2023-09-21T13:22:35.496 INFO:teuthology.lock.ops:unlocked: smithi045.front.sepia.ceph.com
2023-09-21T13:22:47.076 DEBUG:teuthology.orchestra.console.smithi110:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T13:22:47.076 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T13:22:52.698 WARNING:teuthology.provision.fog.smithi045:[Errno None] Unable to connect to port 22 on 172.21.15.45
2023-09-21T13:22:58.706 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T13:23:01.786 WARNING:teuthology.provision.fog.smithi045:[Errno None] Unable to connect to port 22 on 172.21.15.45
2023-09-21T13:23:07.149 DEBUG:teuthology.orchestra.console.smithi110:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T13:23:07.150 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T13:23:07.789 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T13:23:10.842 WARNING:teuthology.provision.fog.smithi045:[Errno None] Unable to connect to port 22 on 172.21.15.45
2023-09-21T13:23:16.850 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T13:23:19.930 WARNING:teuthology.provision.fog.smithi045:[Errno None] Unable to connect to port 22 on 172.21.15.45
2023-09-21T13:23:25.935 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T13:23:27.223 DEBUG:teuthology.orchestra.console.smithi110:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T13:23:27.224 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:23:28.232 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:23:28.986 WARNING:teuthology.provision.fog.smithi045:[Errno None] Unable to connect to port 22 on 172.21.15.45
2023-09-21T13:23:32.234 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:23:33.345 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:23:34.990 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T13:23:37.349 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:23:38.042 WARNING:teuthology.provision.fog.smithi045:[Errno None] Unable to connect to port 22 on 172.21.15.45
2023-09-21T13:23:38.459 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:23:42.462 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:23:43.572 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:23:44.045 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T13:23:47.098 WARNING:teuthology.provision.fog.smithi045:[Errno None] Unable to connect to port 22 on 172.21.15.45
2023-09-21T13:23:47.574 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:23:48.685 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:23:52.690 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:23:53.101 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T13:23:53.801 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:23:56.154 WARNING:teuthology.provision.fog.smithi045:[Errno None] Unable to connect to port 22 on 172.21.15.45
2023-09-21T13:23:57.804 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:23:58.913 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:24:02.159 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T13:24:02.915 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:24:04.029 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:24:05.210 WARNING:teuthology.provision.fog.smithi045:[Errno None] Unable to connect to port 22 on 172.21.15.45
2023-09-21T13:24:08.033 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:24:09.142 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:24:11.214 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T13:24:13.145 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:24:14.255 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:24:18.258 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:24:19.367 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:24:23.370 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:24:24.479 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:24:28.482 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:24:29.590 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:24:33.594 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:24:34.703 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:24:38.706 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T13:24:39.814 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T13:24:39.815 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T13:24:39.815 ERROR:teuthology.orchestra.console.smithi110:Failed to power off
2023-09-21T13:24:39.962 INFO:teuthology.lock.ops:unlocked: smithi110.front.sepia.ceph.com
2023-09-21T13:24:40.152 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi110