2023-09-21T10:32:51.522 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-21T10:32:51.523 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-09-21T10:32:51.697 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7fef5e518d30>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7fef5e1c3ca0>, signals=[15])
2023-09-21T10:32:51.699 INFO:teuthology.lock.ops:Start node 'smithi071.front.sepia.ceph.com' reimaging
2023-09-21T10:32:51.699 INFO:teuthology.lock.ops:Updating [smithi071.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T10:32:51.699 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server
2023-09-21T10:32:51.722 INFO:teuthology.lock.ops:Start node 'smithi098.front.sepia.ceph.com' reimaging
2023-09-21T10:32:51.723 INFO:teuthology.lock.ops:Updating [smithi098.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T10:32:51.723 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server
2023-09-21T10:32:51.724 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /host {"name": "smithi071"}
2023-09-21T10:32:51.748 DEBUG:teuthology.provision.fog.smithi098:FOG request: GET /host {"name": "smithi098"}
2023-09-21T10:32:51.799 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /image {"name": "smithi_centos_9.stream"}
2023-09-21T10:32:51.817 DEBUG:teuthology.provision.fog.smithi098:FOG request: GET /image {"name": "smithi_centos_9.stream"}
2023-09-21T10:32:51.839 DEBUG:teuthology.provision.fog.smithi071:Requesting image smithi_centos_9.stream (ID 10)
2023-09-21T10:32:51.839 DEBUG:teuthology.provision.fog.smithi071:FOG request: PUT /host/176 {"imageID": 10}
2023-09-21T10:32:51.860 DEBUG:teuthology.provision.fog.smithi098:Requesting image smithi_centos_9.stream (ID 10)
2023-09-21T10:32:51.860 DEBUG:teuthology.provision.fog.smithi098:FOG request: PUT /host/203 {"imageID": 10}
2023-09-21T10:32:51.885 INFO:teuthology.provision.fog.smithi071:Scheduling deploy of centos 9.stream
2023-09-21T10:32:51.885 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /task/active None
2023-09-21T10:32:51.905 INFO:teuthology.provision.fog.smithi098:Scheduling deploy of centos 9.stream
2023-09-21T10:32:51.905 DEBUG:teuthology.provision.fog.smithi098:FOG request: GET /task/active None
2023-09-21T10:32:51.946 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T10:32:51.953 DEBUG:teuthology.provision.fog.smithi098:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T10:32:51.982 DEBUG:teuthology.provision.fog.smithi071:FOG request: POST /host/176/task {"taskTypeID": 1}
2023-09-21T10:32:51.988 DEBUG:teuthology.provision.fog.smithi098:FOG request: POST /host/203/task {"taskTypeID": 1}
2023-09-21T10:32:52.041 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /task/active None
2023-09-21T10:32:52.045 DEBUG:teuthology.provision.fog.smithi098:FOG request: GET /task/active None
2023-09-21T10:32:52.135 INFO:teuthology.orchestra.console.smithi071:Power off
2023-09-21T10:32:52.136 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T10:32:52.149 INFO:teuthology.orchestra.console.smithi098:Power off
2023-09-21T10:32:52.150 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T10:32:52.159 DEBUG:teuthology.orchestra.console.smithi071:power off output: Chassis Power Control: Down/Off
2023-09-21T10:32:52.159 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:32:52.184 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on
2023-09-21T10:32:56.186 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:32:56.297 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on
2023-09-21T10:33:00.302 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:33:00.411 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is off
2023-09-21T10:33:00.512 INFO:teuthology.orchestra.console.smithi071:Power off completed
2023-09-21T10:33:00.613 INFO:teuthology.orchestra.console.smithi071:Power on
2023-09-21T10:33:00.613 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T10:33:00.637 DEBUG:teuthology.orchestra.console.smithi071:power on output: Chassis Power Control: Up/On
2023-09-21T10:33:00.637 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:33:00.660 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is off
2023-09-21T10:33:04.663 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:33:04.772 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on
2023-09-21T10:33:04.873 INFO:teuthology.orchestra.console.smithi071:Power on completed
2023-09-21T10:33:04.974 INFO:teuthology.provision.fog.smithi071:Waiting for deploy to finish
2023-09-21T10:33:04.974 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /task/active None
2023-09-21T10:33:12.219 DEBUG:teuthology.orchestra.console.smithi098:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T10:33:12.220 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T10:33:20.058 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /task/active None
2023-09-21T10:33:32.287 DEBUG:teuthology.orchestra.console.smithi098:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T10:33:32.288 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T10:33:35.130 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /task/active None
2023-09-21T10:33:50.219 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /task/active None
2023-09-21T10:33:52.355 DEBUG:teuthology.orchestra.console.smithi098:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T10:33:52.356 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:33:53.364 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:33:57.366 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:33:58.474 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:34:02.478 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:34:03.586 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:34:05.281 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /task/active None
2023-09-21T10:34:07.589 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:34:08.697 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:34:12.702 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:34:13.811 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:34:17.814 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:34:18.924 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:34:20.347 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /task/active None
2023-09-21T10:34:22.928 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:34:24.037 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:34:28.042 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:34:29.152 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:34:33.156 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:34:34.265 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:34:35.414 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /task/active None
2023-09-21T10:34:38.269 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:34:39.377 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:34:43.382 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:34:44.491 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:34:48.495 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:34:49.604 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:34:50.487 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /task/active None
2023-09-21T10:34:53.609 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:34:54.717 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:34:58.722 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:34:59.831 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:35:03.834 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:35:04.944 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:35:04.945 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T10:35:04.945 ERROR:teuthology.orchestra.console.smithi098:Failed to power off
2023-09-21T10:35:04.946 INFO:teuthology.orchestra.console.smithi098:Power on
2023-09-21T10:35:04.946 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T10:35:05.561 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /task/active None
2023-09-21T10:35:20.638 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /task/active None
2023-09-21T10:35:25.016 DEBUG:teuthology.orchestra.console.smithi098:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T10:35:25.016 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T10:35:35.716 DEBUG:teuthology.provision.fog.smithi071:FOG request: GET /task/active None
2023-09-21T10:35:35.782 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T10:35:45.089 DEBUG:teuthology.orchestra.console.smithi098:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T10:35:45.089 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T10:35:54.138 WARNING:teuthology.provision.fog.smithi071:[Errno None] Unable to connect to port 22 on 172.21.15.71
2023-09-21T10:36:00.146 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T10:36:03.226 WARNING:teuthology.provision.fog.smithi071:[Errno None] Unable to connect to port 22 on 172.21.15.71
2023-09-21T10:36:05.161 DEBUG:teuthology.orchestra.console.smithi098:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T10:36:05.162 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:36:06.173 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:36:09.231 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T10:36:10.176 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:36:11.284 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:36:12.282 WARNING:teuthology.provision.fog.smithi071:[Errno None] Unable to connect to port 22 on 172.21.15.71
2023-09-21T10:36:15.287 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:36:16.396 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:36:18.285 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T10:36:20.398 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:36:21.342 WARNING:teuthology.provision.fog.smithi071:[Errno None] Unable to connect to port 22 on 172.21.15.71
2023-09-21T10:36:21.506 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:36:25.510 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:36:26.618 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:36:27.345 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T10:36:30.426 WARNING:teuthology.provision.fog.smithi071:[Errno None] Unable to connect to port 22 on 172.21.15.71
2023-09-21T10:36:30.620 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:36:31.729 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:36:35.734 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:36:36.429 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T10:36:36.842 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:36:40.846 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:36:41.954 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:36:45.958 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:36:47.066 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:36:51.070 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:36:52.178 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:36:56.182 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:36:57.290 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:37:01.294 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:37:02.403 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:37:06.406 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:37:07.514 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:37:11.517 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:37:12.626 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:37:16.630 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:37:17.739 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:37:17.739 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T10:37:17.740 DEBUG:teuthology.provision.fog.smithi098:Canceling deploy task with ID 499142
2023-09-21T10:37:17.740 DEBUG:teuthology.provision.fog.smithi098:FOG request: DELETE /task/cancel {"id": 499142}
2023-09-21T10:37:17.792 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi098
2023-09-21T10:37:17.793 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-09-21T10:37:17.929 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=a676a19a22584463bd97faf1d7252db4
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi098
2023-09-21T10:37:17.931 INFO:teuthology.nuke:Checking targets against current locks
2023-09-21T10:37:17.996 DEBUG:teuthology.nuke:shortname: smithi071
2023-09-21T10:37:17.996 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T10:37:18.009 DEBUG:teuthology.nuke:shortname: smithi098
2023-09-21T10:37:18.009 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T10:37:18.015 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mkogan-2023-09-20_11:11:11-rgw-rgw-wip-t62141-valg-supp_i007-distro-default-smithi/7399426', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 10:32:50.711766', 'locked_by': 'scheduled_mkogan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP+kn06Ap8Aqmpry6U6dEvck4yh/NTyyLvgsRO9IP4T2cLtBIz0fVto8MZZq+nqbX9U0jkdqVspO9Z8K/drWSAU='}
2023-09-21T10:37:18.031 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mkogan-2023-09-20_11:11:11-rgw-rgw-wip-t62141-valg-supp_i007-distro-default-smithi/7399426', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 10:32:50.713226', 'locked_by': 'scheduled_mkogan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA7O8m0rgonywG4rM5Kk/eydCwxQanlLT96+GYOYAXp72L0Rg5MIENHndfH9ozKM5CTGrEvPEa6LyVyXlesSpmg='}
2023-09-21T10:37:18.058 INFO:teuthology.orchestra.console.smithi071:Power off
2023-09-21T10:37:18.059 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T10:37:18.081 DEBUG:teuthology.orchestra.console.smithi071:power off output: Chassis Power Control: Down/Off
2023-09-21T10:37:18.082 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:37:18.095 INFO:teuthology.orchestra.console.smithi098:Power off
2023-09-21T10:37:18.095 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T10:37:18.103 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on
2023-09-21T10:37:22.107 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:37:22.216 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on
2023-09-21T10:37:26.218 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:37:26.327 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is off
2023-09-21T10:37:26.428 INFO:teuthology.orchestra.console.smithi071:Power off completed
2023-09-21T10:37:26.582 INFO:teuthology.lock.ops:unlocked: smithi071.front.sepia.ceph.com
2023-09-21T10:37:36.438 WARNING:teuthology.provision.fog.smithi071:timed out
2023-09-21T10:37:38.163 DEBUG:teuthology.orchestra.console.smithi098:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T10:37:38.164 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T10:37:42.443 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T10:37:58.233 DEBUG:teuthology.orchestra.console.smithi098:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T10:37:58.234 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T10:38:00.858 WARNING:teuthology.provision.fog.smithi071:[Errno None] Unable to connect to port 22 on 172.21.15.71
2023-09-21T10:38:06.862 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T10:38:09.914 WARNING:teuthology.provision.fog.smithi071:[Errno None] Unable to connect to port 22 on 172.21.15.71
2023-09-21T10:38:15.921 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T10:38:18.303 DEBUG:teuthology.orchestra.console.smithi098:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T10:38:18.304 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:38:19.002 WARNING:teuthology.provision.fog.smithi071:[Errno None] Unable to connect to port 22 on 172.21.15.71
2023-09-21T10:38:19.313 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:38:23.315 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:38:24.423 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:38:25.005 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T10:38:28.062 WARNING:teuthology.provision.fog.smithi071:[Errno None] Unable to connect to port 22 on 172.21.15.71
2023-09-21T10:38:28.425 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:38:29.534 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:38:33.539 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:38:34.065 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T10:38:34.650 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:38:37.146 WARNING:teuthology.provision.fog.smithi071:[Errno None] Unable to connect to port 22 on 172.21.15.71
2023-09-21T10:38:38.652 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:38:39.761 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:38:43.152 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T10:38:43.762 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:38:44.872 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:38:46.234 WARNING:teuthology.provision.fog.smithi071:[Errno None] Unable to connect to port 22 on 172.21.15.71
2023-09-21T10:38:48.876 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:38:49.985 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:38:52.236 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T10:38:53.988 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:38:55.097 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:38:59.102 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:39:00.211 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:39:04.214 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:39:05.323 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:39:09.326 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:39:10.434 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:39:10.746 WARNING:teuthology.provision.fog.smithi071:[Errno None] Unable to connect to port 22 on 172.21.15.71
2023-09-21T10:39:14.438 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:39:15.546 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:39:16.750 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T10:39:16.874 ERROR:teuthology.orchestra.connection:Error authenticating with smithi071.front.sepia.ceph.com: Authentication failed.
2023-09-21T10:39:16.877 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-09-21T10:39:16.878 DEBUG:teuthology.orchestra.run.smithi071:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-09-21T10:39:16.881 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T10:39:17.887 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T10:39:18.893 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T10:39:19.547 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:39:19.901 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T10:39:20.656 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:39:20.908 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T10:39:21.914 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T10:39:22.919 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T10:39:23.925 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T10:39:24.657 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:39:24.932 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T10:39:25.766 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:39:25.938 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T10:39:26.944 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T10:39:27.950 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T10:39:28.955 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T10:39:29.768 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T10:39:29.962 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T10:39:30.877 DEBUG:teuthology.orchestra.console.smithi098:check power output: 
2023-09-21T10:39:30.877 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T10:39:30.877 ERROR:teuthology.orchestra.console.smithi098:Failed to power off
2023-09-21T10:39:30.968 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T10:39:31.031 INFO:teuthology.lock.ops:unlocked: smithi098.front.sepia.ceph.com
2023-09-21T10:39:31.281 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi098