2023-09-21T12:10:24.281 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-21T12:10:24.282 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-09-21T12:10:24.438 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7fa768beccd0>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7fa7688a1c10>, signals=[15])
2023-09-21T12:10:24.439 INFO:teuthology.lock.ops:Start node 'smithi112.front.sepia.ceph.com' reimaging
2023-09-21T12:10:24.439 INFO:teuthology.lock.ops:Updating [smithi112.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T12:10:24.440 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server
2023-09-21T12:10:24.461 INFO:teuthology.lock.ops:Start node 'smithi138.front.sepia.ceph.com' reimaging
2023-09-21T12:10:24.462 INFO:teuthology.lock.ops:Updating [smithi138.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T12:10:24.462 INFO:teuthology.lock.ops:Updating smithi138.front.sepia.ceph.com on lock server
2023-09-21T12:10:24.463 DEBUG:teuthology.provision.fog.smithi112:FOG request: GET /host {"name": "smithi112"}
2023-09-21T12:10:24.486 DEBUG:teuthology.provision.fog.smithi138:FOG request: GET /host {"name": "smithi138"}
2023-09-21T12:10:24.547 DEBUG:teuthology.provision.fog.smithi112:FOG request: GET /image {"name": "smithi_centos_8.stream"}
2023-09-21T12:10:24.551 DEBUG:teuthology.provision.fog.smithi138:FOG request: GET /image {"name": "smithi_centos_8.stream"}
2023-09-21T12:10:24.589 DEBUG:teuthology.provision.fog.smithi112:Requesting image smithi_centos_8.stream (ID 3)
2023-09-21T12:10:24.590 DEBUG:teuthology.provision.fog.smithi112:FOG request: PUT /host/217 {"imageID": 3}
2023-09-21T12:10:24.600 DEBUG:teuthology.provision.fog.smithi138:Requesting image smithi_centos_8.stream (ID 3)
2023-09-21T12:10:24.600 DEBUG:teuthology.provision.fog.smithi138:FOG request: PUT /host/243 {"imageID": 3}
2023-09-21T12:10:24.634 INFO:teuthology.provision.fog.smithi112:Scheduling deploy of centos 8.stream
2023-09-21T12:10:24.634 DEBUG:teuthology.provision.fog.smithi112:FOG request: GET /task/active None
2023-09-21T12:10:24.642 INFO:teuthology.provision.fog.smithi138:Scheduling deploy of centos 8.stream
2023-09-21T12:10:24.642 DEBUG:teuthology.provision.fog.smithi138:FOG request: GET /task/active None
2023-09-21T12:10:24.703 DEBUG:teuthology.provision.fog.smithi112:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T12:10:24.705 DEBUG:teuthology.provision.fog.smithi138:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T12:10:24.746 DEBUG:teuthology.provision.fog.smithi138:FOG request: POST /host/243/task {"taskTypeID": 1}
2023-09-21T12:10:24.749 DEBUG:teuthology.provision.fog.smithi112:FOG request: POST /host/217/task {"taskTypeID": 1}
2023-09-21T12:10:24.809 DEBUG:teuthology.provision.fog.smithi112:FOG request: GET /task/active None
2023-09-21T12:10:24.813 DEBUG:teuthology.provision.fog.smithi138:FOG request: GET /task/active None
2023-09-21T12:10:24.905 INFO:teuthology.orchestra.console.smithi112:Power off
2023-09-21T12:10:24.906 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:10:24.919 INFO:teuthology.orchestra.console.smithi138:Power off
2023-09-21T12:10:24.920 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:10:24.941 DEBUG:teuthology.orchestra.console.smithi138:power off output: Chassis Power Control: Down/Off
2023-09-21T12:10:24.942 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:10:24.966 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is off
2023-09-21T12:10:25.067 INFO:teuthology.orchestra.console.smithi138:Power off completed
2023-09-21T12:10:25.168 INFO:teuthology.orchestra.console.smithi138:Power on
2023-09-21T12:10:25.168 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T12:10:25.190 DEBUG:teuthology.orchestra.console.smithi138:power on output: Chassis Power Control: Up/On
2023-09-21T12:10:25.191 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:10:25.211 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is off
2023-09-21T12:10:29.214 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:10:29.323 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is off
2023-09-21T12:10:33.326 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:10:33.434 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is on
2023-09-21T12:10:33.535 INFO:teuthology.orchestra.console.smithi138:Power on completed
2023-09-21T12:10:33.636 INFO:teuthology.provision.fog.smithi138:Waiting for deploy to finish
2023-09-21T12:10:33.637 DEBUG:teuthology.provision.fog.smithi138:FOG request: GET /task/active None
2023-09-21T12:10:44.972 DEBUG:teuthology.orchestra.console.smithi112:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:10:44.973 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:10:48.718 DEBUG:teuthology.provision.fog.smithi138:FOG request: GET /task/active None
2023-09-21T12:11:03.798 DEBUG:teuthology.provision.fog.smithi138:FOG request: GET /task/active None
2023-09-21T12:11:05.043 DEBUG:teuthology.orchestra.console.smithi112:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:11:05.044 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:11:18.870 DEBUG:teuthology.provision.fog.smithi138:FOG request: GET /task/active None
2023-09-21T12:11:25.112 DEBUG:teuthology.orchestra.console.smithi112:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:11:25.112 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:11:26.121 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:11:30.124 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:11:31.233 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:11:33.970 DEBUG:teuthology.provision.fog.smithi138:FOG request: GET /task/active None
2023-09-21T12:11:35.236 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:11:36.345 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:11:40.350 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:11:41.459 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:11:45.462 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:11:46.570 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:11:49.047 DEBUG:teuthology.provision.fog.smithi138:FOG request: GET /task/active None
2023-09-21T12:11:50.572 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:11:51.681 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:11:55.686 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:11:56.794 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:12:00.798 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:12:01.906 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:12:04.131 DEBUG:teuthology.provision.fog.smithi138:FOG request: GET /task/active None
2023-09-21T12:12:05.908 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:12:07.016 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:12:11.021 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:12:12.130 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:12:16.134 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:12:17.243 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:12:19.211 DEBUG:teuthology.provision.fog.smithi138:FOG request: GET /task/active None
2023-09-21T12:12:21.246 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:12:22.354 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:12:26.358 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:12:27.467 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:12:31.470 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:12:32.578 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:12:34.296 DEBUG:teuthology.provision.fog.smithi138:FOG request: GET /task/active None
2023-09-21T12:12:36.581 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:12:37.690 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:12:37.691 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T12:12:37.691 ERROR:teuthology.orchestra.console.smithi112:Failed to power off
2023-09-21T12:12:37.692 INFO:teuthology.orchestra.console.smithi112:Power on
2023-09-21T12:12:37.692 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T12:12:49.366 DEBUG:teuthology.provision.fog.smithi138:FOG request: GET /task/active None
2023-09-21T12:12:57.763 DEBUG:teuthology.orchestra.console.smithi112:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:12:57.764 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T12:13:04.446 DEBUG:teuthology.provision.fog.smithi138:FOG request: GET /task/active None
2023-09-21T12:13:17.832 DEBUG:teuthology.orchestra.console.smithi112:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:13:17.832 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T12:13:19.521 DEBUG:teuthology.provision.fog.smithi138:FOG request: GET /task/active None
2023-09-21T12:13:19.608 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:13:37.902 DEBUG:teuthology.orchestra.console.smithi112:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:13:37.903 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:13:38.074 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138
2023-09-21T12:13:38.910 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:13:42.914 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:13:44.022 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:13:44.076 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:13:47.130 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138
2023-09-21T12:13:48.024 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:13:49.133 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:13:53.134 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:13:53.136 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:13:54.245 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:13:56.186 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138
2023-09-21T12:13:58.249 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:13:59.357 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:14:02.190 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:14:03.359 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:14:04.469 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:14:05.242 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138
2023-09-21T12:14:08.474 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:14:09.583 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:14:11.246 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:14:13.586 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:14:14.694 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:14:18.698 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:14:19.807 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:14:23.810 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:14:24.919 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:14:28.923 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:14:30.032 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:14:34.036 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:14:35.145 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:14:39.150 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:14:40.258 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:14:44.262 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:14:45.370 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:14:49.374 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:14:50.482 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:14:50.483 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T12:14:50.483 DEBUG:teuthology.provision.fog.smithi112:Canceling deploy task with ID 499469
2023-09-21T12:14:50.483 DEBUG:teuthology.provision.fog.smithi112:FOG request: DELETE /task/cancel {"id": 499469}
2023-09-21T12:14:50.540 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi112
2023-09-21T12:14:50.540 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-09-21T12:14:50.699 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ab57aff8896748278caf5f509715a005
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi112
2023-09-21T12:14:50.702 INFO:teuthology.nuke:Checking targets against current locks
2023-09-21T12:14:50.763 DEBUG:teuthology.nuke:shortname: smithi112
2023-09-21T12:14:50.764 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T12:14:50.789 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-09-20_16:32:45-fs-wip-lusov-config-format-distro-default-smithi/7399843', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 12:10:23.463796', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNsN1SFtSMFYxyYNcbgvW1eDZSesLM9jY4HJEMXKIWGOAdf0ieWEX3W/j2HEYjhwPyOGTJiDQ1tGjXWJ9ToC5jc='}
2023-09-21T12:14:50.793 DEBUG:teuthology.nuke:shortname: smithi138
2023-09-21T12:14:50.794 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T12:14:50.817 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi138.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-09-20_16:32:45-fs-wip-lusov-config-format-distro-default-smithi/7399843', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 12:10:23.461852', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPXMpzF69eTJFvUvyFBroJM8I9U74jXltIxp9CLUwa6v9acbNn28zuxrXScLmjpLfPwXY61CrpmDWfRvu9ZQb3w='}
2023-09-21T12:14:50.850 INFO:teuthology.orchestra.console.smithi112:Power off
2023-09-21T12:14:50.850 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:14:50.863 INFO:teuthology.orchestra.console.smithi138:Power off
2023-09-21T12:14:50.864 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:14:50.885 DEBUG:teuthology.orchestra.console.smithi138:power off output: Chassis Power Control: Down/Off
2023-09-21T12:14:50.886 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:14:50.907 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is on
2023-09-21T12:14:54.910 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:14:55.019 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is on
2023-09-21T12:14:59.022 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:14:59.132 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is off
2023-09-21T12:14:59.233 INFO:teuthology.orchestra.console.smithi138:Power off completed
2023-09-21T12:14:59.402 INFO:teuthology.lock.ops:unlocked: smithi138.front.sepia.ceph.com
2023-09-21T12:15:10.920 DEBUG:teuthology.orchestra.console.smithi112:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:15:10.920 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:15:11.249 WARNING:teuthology.provision.fog.smithi138:timed out
2023-09-21T12:15:17.256 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:15:30.988 DEBUG:teuthology.orchestra.console.smithi112:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:15:30.988 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T12:15:35.582 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138
2023-09-21T12:15:41.587 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:15:44.666 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138
2023-09-21T12:15:50.670 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:15:51.062 DEBUG:teuthology.orchestra.console.smithi112:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T12:15:51.064 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:15:52.074 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:15:53.726 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138
2023-09-21T12:15:56.076 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:15:57.185 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:15:59.730 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:16:01.188 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:16:02.298 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:16:02.810 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138
2023-09-21T12:16:06.302 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:16:07.410 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:16:08.813 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:16:11.414 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:16:11.866 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138
2023-09-21T12:16:12.523 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:16:16.526 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:16:17.635 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:16:17.868 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:16:20.922 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138
2023-09-21T12:16:21.638 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:16:22.748 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:16:26.752 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:16:26.924 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:16:27.862 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:16:31.866 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:16:32.978 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:16:36.982 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:16:37.278 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138
2023-09-21T12:16:38.091 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:16:42.094 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:16:43.204 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:16:43.280 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:16:46.362 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138
2023-09-21T12:16:47.207 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:16:48.316 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:16:52.318 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:16:52.364 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T12:16:52.485 ERROR:teuthology.orchestra.connection:Error authenticating with smithi138.front.sepia.ceph.com: Authentication failed.
2023-09-21T12:16:52.489 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-09-21T12:16:52.490 DEBUG:teuthology.orchestra.run.smithi138:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-09-21T12:16:52.492 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:16:53.427 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:16:53.498 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:16:54.504 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:16:55.510 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:16:56.517 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:16:57.427 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:16:57.524 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:16:58.531 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:16:58.537 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:16:59.538 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:17:00.544 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:17:01.551 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:17:02.538 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T12:17:02.558 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:17:03.565 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T12:17:03.649 DEBUG:teuthology.orchestra.console.smithi112:check power output: 
2023-09-21T12:17:03.650 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T12:17:03.651 ERROR:teuthology.orchestra.console.smithi112:Failed to power off
2023-09-21T12:17:03.813 INFO:teuthology.lock.ops:unlocked: smithi112.front.sepia.ceph.com
2023-09-21T12:17:04.097 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi112