2023-09-21T14:00:29.343 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-21T14:00:29.345 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-09-21T14:00:29.487 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7fad613d5d60>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7fad61ec6670>, signals=[15])
2023-09-21T14:00:29.488 INFO:teuthology.lock.ops:Start node 'smithi053.front.sepia.ceph.com' reimaging
2023-09-21T14:00:29.489 INFO:teuthology.lock.ops:Updating [smithi053.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T14:00:29.489 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server
2023-09-21T14:00:29.510 INFO:teuthology.lock.ops:Start node 'smithi110.front.sepia.ceph.com' reimaging
2023-09-21T14:00:29.511 INFO:teuthology.lock.ops:Updating [smithi110.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T14:00:29.511 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server
2023-09-21T14:00:29.512 DEBUG:teuthology.provision.fog.smithi053:FOG request: GET /host {"name": "smithi053"}
2023-09-21T14:00:29.535 DEBUG:teuthology.provision.fog.smithi110:FOG request: GET /host {"name": "smithi110"}
2023-09-21T14:00:29.590 DEBUG:teuthology.provision.fog.smithi053:FOG request: GET /image {"name": "smithi_ubuntu_20.04"}
2023-09-21T14:00:29.607 DEBUG:teuthology.provision.fog.smithi110:FOG request: GET /image {"name": "smithi_ubuntu_20.04"}
2023-09-21T14:00:29.633 DEBUG:teuthology.provision.fog.smithi053:Requesting image smithi_ubuntu_20.04 (ID 2)
2023-09-21T14:00:29.634 DEBUG:teuthology.provision.fog.smithi053:FOG request: PUT /host/158 {"imageID": 2}
2023-09-21T14:00:29.651 DEBUG:teuthology.provision.fog.smithi110:Requesting image smithi_ubuntu_20.04 (ID 2)
2023-09-21T14:00:29.651 DEBUG:teuthology.provision.fog.smithi110:FOG request: PUT /host/215 {"imageID": 2}
2023-09-21T14:00:29.680 INFO:teuthology.provision.fog.smithi053:Scheduling deploy of ubuntu 20.04
2023-09-21T14:00:29.680 DEBUG:teuthology.provision.fog.smithi053:FOG request: GET /task/active None
2023-09-21T14:00:29.700 INFO:teuthology.provision.fog.smithi110:Scheduling deploy of ubuntu 20.04
2023-09-21T14:00:29.700 DEBUG:teuthology.provision.fog.smithi110:FOG request: GET /task/active None
2023-09-21T14:00:29.773 DEBUG:teuthology.provision.fog.smithi053:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T14:00:29.784 DEBUG:teuthology.provision.fog.smithi110:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T14:00:29.807 DEBUG:teuthology.provision.fog.smithi053:FOG request: POST /host/158/task {"taskTypeID": 1}
2023-09-21T14:00:29.820 DEBUG:teuthology.provision.fog.smithi110:FOG request: POST /host/215/task {"taskTypeID": 1}
2023-09-21T14:00:29.855 DEBUG:teuthology.provision.fog.smithi053:FOG request: GET /task/active None
2023-09-21T14:00:29.865 DEBUG:teuthology.provision.fog.smithi110:FOG request: GET /task/active None
2023-09-21T14:00:29.986 INFO:teuthology.orchestra.console.smithi110:Power off
2023-09-21T14:00:29.987 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T14:00:30.000 INFO:teuthology.orchestra.console.smithi053:Power off
2023-09-21T14:00:30.000 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T14:00:30.021 DEBUG:teuthology.orchestra.console.smithi053:power off output: Chassis Power Control: Down/Off
2023-09-21T14:00:30.021 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:00:30.042 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off
2023-09-21T14:00:30.143 INFO:teuthology.orchestra.console.smithi053:Power off completed
2023-09-21T14:00:30.244 INFO:teuthology.orchestra.console.smithi053:Power on
2023-09-21T14:00:30.245 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T14:00:30.265 DEBUG:teuthology.orchestra.console.smithi053:power on output: Chassis Power Control: Up/On
2023-09-21T14:00:30.266 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:00:30.288 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off
2023-09-21T14:00:34.290 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:00:34.399 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on
2023-09-21T14:00:34.500 INFO:teuthology.orchestra.console.smithi053:Power on completed
2023-09-21T14:00:34.601 INFO:teuthology.provision.fog.smithi053:Waiting for deploy to finish
2023-09-21T14:00:34.602 DEBUG:teuthology.provision.fog.smithi053:FOG request: GET /task/active None
2023-09-21T14:00:49.702 DEBUG:teuthology.provision.fog.smithi053:FOG request: GET /task/active None
2023-09-21T14:00:50.056 DEBUG:teuthology.orchestra.console.smithi110:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T14:00:50.056 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T14:01:04.798 DEBUG:teuthology.provision.fog.smithi053:FOG request: GET /task/active None
2023-09-21T14:01:10.132 DEBUG:teuthology.orchestra.console.smithi110:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T14:01:10.133 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T14:01:19.902 DEBUG:teuthology.provision.fog.smithi053:FOG request: GET /task/active None
2023-09-21T14:01:30.205 DEBUG:teuthology.orchestra.console.smithi110:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T14:01:30.206 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:01:31.214 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:01:34.994 DEBUG:teuthology.provision.fog.smithi053:FOG request: GET /task/active None
2023-09-21T14:01:35.215 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:01:36.325 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:01:40.330 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:01:41.439 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:01:45.444 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:01:46.552 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:01:50.083 DEBUG:teuthology.provision.fog.smithi053:FOG request: GET /task/active None
2023-09-21T14:01:50.553 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:01:51.662 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:01:55.667 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:01:56.775 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:02:00.778 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:02:01.886 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:02:05.182 DEBUG:teuthology.provision.fog.smithi053:FOG request: GET /task/active None
2023-09-21T14:02:05.887 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:02:06.996 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:02:10.998 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:02:12.106 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:02:16.110 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:02:17.219 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:02:20.270 DEBUG:teuthology.provision.fog.smithi053:FOG request: GET /task/active None
2023-09-21T14:02:21.221 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:02:22.329 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:02:26.334 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:02:27.442 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:02:31.446 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:02:32.554 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:02:35.352 DEBUG:teuthology.provision.fog.smithi053:FOG request: GET /task/active None
2023-09-21T14:02:36.556 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:02:37.664 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:02:41.666 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:02:42.774 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:02:42.775 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T14:02:42.775 ERROR:teuthology.orchestra.console.smithi110:Failed to power off
2023-09-21T14:02:42.775 INFO:teuthology.orchestra.console.smithi110:Power on
2023-09-21T14:02:42.776 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T14:02:50.438 DEBUG:teuthology.provision.fog.smithi053:FOG request: GET /task/active None
2023-09-21T14:03:02.845 DEBUG:teuthology.orchestra.console.smithi110:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T14:03:02.845 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T14:03:05.522 DEBUG:teuthology.provision.fog.smithi053:FOG request: GET /task/active None
2023-09-21T14:03:20.611 DEBUG:teuthology.provision.fog.smithi053:FOG request: GET /task/active None
2023-09-21T14:03:22.912 DEBUG:teuthology.orchestra.console.smithi110:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T14:03:22.912 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T14:03:35.711 DEBUG:teuthology.provision.fog.smithi053:FOG request: GET /task/active None
2023-09-21T14:03:42.981 DEBUG:teuthology.orchestra.console.smithi110:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T14:03:42.981 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:03:43.990 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:03:47.994 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:03:49.103 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:03:50.793 DEBUG:teuthology.provision.fog.smithi053:FOG request: GET /task/active None
2023-09-21T14:03:53.107 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:03:54.216 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:03:58.221 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:03:59.330 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:04:03.349 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:04:04.458 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:04:05.892 DEBUG:teuthology.provision.fog.smithi053:FOG request: GET /task/active None
2023-09-21T14:04:05.989 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:04:08.461 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:04:09.569 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:04:13.574 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:04:14.682 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:04:18.686 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:04:19.795 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:04:23.800 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:04:24.346 WARNING:teuthology.provision.fog.smithi053:[Errno None] Unable to connect to port 22 on 172.21.15.53
2023-09-21T14:04:24.908 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:04:28.910 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:04:30.018 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:04:30.349 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:04:33.402 WARNING:teuthology.provision.fog.smithi053:[Errno None] Unable to connect to port 22 on 172.21.15.53
2023-09-21T14:04:34.023 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:04:35.134 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:04:39.139 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:04:39.404 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:04:40.248 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:04:42.462 WARNING:teuthology.provision.fog.smithi053:[Errno None] Unable to connect to port 22 on 172.21.15.53
2023-09-21T14:04:44.250 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:04:45.358 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:04:48.467 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:04:49.361 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:04:50.469 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:04:54.474 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:04:55.582 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:04:55.583 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T14:04:55.584 DEBUG:teuthology.provision.fog.smithi110:Canceling deploy task with ID 499907
2023-09-21T14:04:55.584 DEBUG:teuthology.provision.fog.smithi110:FOG request: DELETE /task/cancel {"id": 499907}
2023-09-21T14:04:55.636 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi110
2023-09-21T14:04:55.637 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-09-21T14:04:55.779 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=5bba2ee26f3b4dc78b9d76804d52e014
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi110
2023-09-21T14:04:55.781 INFO:teuthology.nuke:Checking targets against current locks
2023-09-21T14:04:55.845 DEBUG:teuthology.nuke:shortname: smithi053
2023-09-21T14:04:55.845 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T14:04:55.863 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/svelar-2023-09-20_07:59:56-rados-wip-svelar-testing-1-2023-09-20-1125-distro-default-smithi/7398748', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 14:00:28.544509', 'locked_by': 'scheduled_svelar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDNnL7BvHl9baD82lgfgXd97GxgALz33Wxskx0rbRa990amxkt7ywkMmhg4yIopJ0l9uWMI5kfzNrkBqtu60E6I='}
2023-09-21T14:04:55.868 DEBUG:teuthology.nuke:shortname: smithi110
2023-09-21T14:04:55.868 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T14:04:55.890 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/svelar-2023-09-20_07:59:56-rados-wip-svelar-testing-1-2023-09-20-1125-distro-default-smithi/7398748', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 14:00:28.546448', 'locked_by': 'scheduled_svelar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLH15lhs6mMmDHFGpdBB16wtVR7ydtDsuyvKFjdKxRrfl7qrh+qcKGU5yB0SJo7JVYjbBB/bEzhiFCKoL0P0Fzk='}
2023-09-21T14:04:55.919 INFO:teuthology.orchestra.console.smithi053:Power off
2023-09-21T14:04:55.920 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T14:04:55.941 DEBUG:teuthology.orchestra.console.smithi053:power off output: Chassis Power Control: Down/Off
2023-09-21T14:04:55.941 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:04:55.958 INFO:teuthology.orchestra.console.smithi110:Power off
2023-09-21T14:04:55.959 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T14:04:55.968 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on
2023-09-21T14:04:59.972 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:05:00.081 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off
2023-09-21T14:05:00.182 INFO:teuthology.orchestra.console.smithi053:Power off completed
2023-09-21T14:05:00.329 INFO:teuthology.lock.ops:unlocked: smithi053.front.sepia.ceph.com
2023-09-21T14:05:16.028 DEBUG:teuthology.orchestra.console.smithi110:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T14:05:16.028 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T14:05:36.097 DEBUG:teuthology.orchestra.console.smithi110:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T14:05:36.098 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T14:05:48.482 WARNING:teuthology.provision.fog.smithi053:timed out
2023-09-21T14:05:54.491 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:05:56.165 DEBUG:teuthology.orchestra.console.smithi110:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T14:05:56.166 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:05:57.174 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:06:01.176 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:06:02.286 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:06:06.290 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:06:07.399 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:06:11.402 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:06:12.511 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:06:16.514 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:06:17.624 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:06:21.626 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:06:22.735 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:06:26.738 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:06:27.848 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:06:28.683 WARNING:teuthology.provision.fog.smithi053:[Errno None] Unable to connect to port 22 on 172.21.15.53
2023-09-21T14:06:31.854 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:06:32.962 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:06:34.687 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T14:06:34.813 ERROR:teuthology.orchestra.connection:Error authenticating with smithi053.front.sepia.ceph.com: Authentication failed.
2023-09-21T14:06:34.817 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-09-21T14:06:34.818 DEBUG:teuthology.orchestra.run.smithi053:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-09-21T14:06:34.820 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:06:35.827 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:06:36.834 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:06:36.963 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:06:37.841 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:06:38.073 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:06:38.847 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:06:39.854 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:06:40.860 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:06:41.866 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:06:42.074 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:06:42.873 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:06:43.184 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:06:43.879 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:06:44.885 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:06:45.892 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:06:46.898 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:06:47.185 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:06:47.905 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:06:48.298 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:06:48.911 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:06:49.917 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:06:50.924 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:06:51.930 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:06:52.299 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:06:52.938 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:06:53.409 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:06:53.945 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:06:54.952 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:06:55.958 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:06:56.965 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:06:57.410 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:06:57.972 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:06:58.520 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:06:58.980 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:06:59.987 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:07:00.994 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:07:02.000 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:07:02.522 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:07:03.008 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:07:03.633 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:07:04.017 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:07:05.025 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:07:06.032 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:07:07.038 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:07:07.634 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T14:07:08.046 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:07:08.743 DEBUG:teuthology.orchestra.console.smithi110:check power output: 
2023-09-21T14:07:08.745 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T14:07:08.745 ERROR:teuthology.orchestra.console.smithi110:Failed to power off
2023-09-21T14:07:08.895 INFO:teuthology.lock.ops:unlocked: smithi110.front.sepia.ceph.com
2023-09-21T14:07:09.056 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T14:07:09.137 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi110