2023-09-21T11:04:49.269 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-21T11:04:49.271 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-09-21T11:04:49.434 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f628dd35d00>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f628dadeca0>, signals=[15])
2023-09-21T11:04:49.436 INFO:teuthology.lock.ops:Start node 'smithi121.front.sepia.ceph.com' reimaging
2023-09-21T11:04:49.437 INFO:teuthology.lock.ops:Updating [smithi121.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T11:04:49.437 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server
2023-09-21T11:04:49.462 INFO:teuthology.lock.ops:Start node 'smithi159.front.sepia.ceph.com' reimaging
2023-09-21T11:04:49.463 INFO:teuthology.lock.ops:Updating [smithi159.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T11:04:49.463 INFO:teuthology.lock.ops:Updating smithi159.front.sepia.ceph.com on lock server
2023-09-21T11:04:49.465 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /host {"name": "smithi121"}
2023-09-21T11:04:49.486 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /host {"name": "smithi159"}
2023-09-21T11:04:49.557 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /image {"name": "smithi_ubuntu_20.04"}
2023-09-21T11:04:49.559 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /image {"name": "smithi_ubuntu_20.04"}
2023-09-21T11:04:49.602 DEBUG:teuthology.provision.fog.smithi121:Requesting image smithi_ubuntu_20.04 (ID 2)
2023-09-21T11:04:49.602 DEBUG:teuthology.provision.fog.smithi121:FOG request: PUT /host/226 {"imageID": 2}
2023-09-21T11:04:49.606 DEBUG:teuthology.provision.fog.smithi159:Requesting image smithi_ubuntu_20.04 (ID 2)
2023-09-21T11:04:49.607 DEBUG:teuthology.provision.fog.smithi159:FOG request: PUT /host/264 {"imageID": 2}
2023-09-21T11:04:49.652 INFO:teuthology.provision.fog.smithi159:Scheduling deploy of ubuntu 20.04
2023-09-21T11:04:49.653 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None
2023-09-21T11:04:49.659 INFO:teuthology.provision.fog.smithi121:Scheduling deploy of ubuntu 20.04
2023-09-21T11:04:49.660 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None
2023-09-21T11:04:49.719 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T11:04:49.723 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T11:04:49.758 DEBUG:teuthology.provision.fog.smithi121:FOG request: POST /host/226/task {"taskTypeID": 1}
2023-09-21T11:04:49.761 DEBUG:teuthology.provision.fog.smithi159:FOG request: POST /host/264/task {"taskTypeID": 1}
2023-09-21T11:04:49.809 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None
2023-09-21T11:04:49.817 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None
2023-09-21T11:04:49.926 INFO:teuthology.orchestra.console.smithi159:Power off
2023-09-21T11:04:49.927 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T11:04:49.939 INFO:teuthology.orchestra.console.smithi121:Power off
2023-09-21T11:04:49.940 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T11:04:49.948 DEBUG:teuthology.orchestra.console.smithi159:power off output: Chassis Power Control: Down/Off
2023-09-21T11:04:49.949 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:04:49.968 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is off
2023-09-21T11:04:50.069 INFO:teuthology.orchestra.console.smithi159:Power off completed
2023-09-21T11:04:50.170 INFO:teuthology.orchestra.console.smithi159:Power on
2023-09-21T11:04:50.171 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T11:04:50.191 DEBUG:teuthology.orchestra.console.smithi159:power on output: Chassis Power Control: Up/On
2023-09-21T11:04:50.192 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:04:50.212 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is off
2023-09-21T11:04:54.217 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:04:54.325 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is off
2023-09-21T11:04:58.326 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:04:58.435 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is on
2023-09-21T11:04:58.536 INFO:teuthology.orchestra.console.smithi159:Power on completed
2023-09-21T11:04:58.637 INFO:teuthology.provision.fog.smithi159:Waiting for deploy to finish
2023-09-21T11:04:58.637 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None
2023-09-21T11:05:10.013 DEBUG:teuthology.orchestra.console.smithi121:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T11:05:10.013 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T11:05:13.718 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None
2023-09-21T11:05:28.797 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None
2023-09-21T11:05:30.079 DEBUG:teuthology.orchestra.console.smithi121:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T11:05:30.080 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T11:05:43.894 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None
2023-09-21T11:05:50.149 DEBUG:teuthology.orchestra.console.smithi121:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T11:05:50.149 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:05:51.158 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:05:55.162 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:05:56.270 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:05:58.970 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None
2023-09-21T11:06:00.273 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:06:01.381 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:06:05.386 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:06:06.494 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:06:10.498 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:06:11.608 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:06:14.069 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None
2023-09-21T11:06:15.611 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:06:16.720 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:06:20.722 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:06:21.831 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:06:25.834 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:06:26.943 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:06:29.150 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None
2023-09-21T11:06:30.947 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:06:32.055 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:06:36.058 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:06:37.167 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:06:41.172 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:06:42.280 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:06:44.263 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None
2023-09-21T11:06:46.283 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:06:47.392 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:06:51.395 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:06:52.504 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:06:56.508 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:06:57.617 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:06:59.365 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None
2023-09-21T11:07:01.621 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:07:02.729 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:07:02.730 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T11:07:02.730 ERROR:teuthology.orchestra.console.smithi121:Failed to power off
2023-09-21T11:07:02.731 INFO:teuthology.orchestra.console.smithi121:Power on
2023-09-21T11:07:02.731 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T11:07:14.477 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None
2023-09-21T11:07:22.801 DEBUG:teuthology.orchestra.console.smithi121:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T11:07:22.802 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T11:07:29.566 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None
2023-09-21T11:07:42.870 DEBUG:teuthology.orchestra.console.smithi121:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T11:07:42.871 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T11:07:44.646 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None
2023-09-21T11:07:59.754 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None
2023-09-21T11:08:02.936 DEBUG:teuthology.orchestra.console.smithi121:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T11:08:02.937 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:08:03.946 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:08:07.951 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:08:09.059 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:08:13.062 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:08:14.170 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:08:14.840 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None
2023-09-21T11:08:18.175 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:08:19.284 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:08:23.286 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:08:24.394 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:08:28.398 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:08:29.509 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:08:29.930 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None
2023-09-21T11:08:33.514 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:08:34.623 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:08:38.628 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:08:39.737 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:08:43.738 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:08:44.848 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:08:45.028 DEBUG:teuthology.provision.fog.smithi159:FOG request: GET /task/active None
2023-09-21T11:08:45.110 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:08:48.850 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:08:49.959 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:08:53.962 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:08:55.070 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:08:59.075 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:09:00.184 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:09:03.518 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159
2023-09-21T11:09:04.186 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:09:05.294 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:09:09.298 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:09:09.521 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:09:10.406 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:09:12.602 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159
2023-09-21T11:09:14.409 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:09:15.518 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:09:15.519 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T11:09:15.519 DEBUG:teuthology.provision.fog.smithi121:Canceling deploy task with ID 499232
2023-09-21T11:09:15.519 DEBUG:teuthology.provision.fog.smithi121:FOG request: DELETE /task/cancel {"id": 499232}
2023-09-21T11:09:15.587 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi121
2023-09-21T11:09:15.588 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-09-21T11:09:15.723 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=5aea38004fdb485c93cd145ec3ccb3ac
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi121
2023-09-21T11:09:15.726 INFO:teuthology.nuke:Checking targets against current locks
2023-09-21T11:09:15.788 DEBUG:teuthology.nuke:shortname: smithi121
2023-09-21T11:09:15.789 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T11:09:15.813 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-09-20_16:32:45-fs-wip-lusov-config-format-distro-default-smithi/7399745', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 11:04:48.359406', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1jn1oLBF86q1pnE7RuFXENl5s/X1tespMys9PDEb7ck5yZ2AevLKGrQjEtmL5Zl6NRsF5t8khgq2lSYP9XV8s='}
2023-09-21T11:09:15.816 DEBUG:teuthology.nuke:shortname: smithi159
2023-09-21T11:09:15.817 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T11:09:15.840 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi159.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-09-20_16:32:45-fs-wip-lusov-config-format-distro-default-smithi/7399745', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 11:04:48.357437', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLlHOaIZaB6qxPD4Gs4dmJRN6X7gwqceW22rGA87zuUksT3hFsoMwv7hwRwTasGY0A6n9N4n437SH7psrrOCQ40='}
2023-09-21T11:09:15.863 INFO:teuthology.orchestra.console.smithi121:Power off
2023-09-21T11:09:15.864 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T11:09:15.901 INFO:teuthology.orchestra.console.smithi159:Power off
2023-09-21T11:09:15.902 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T11:09:15.923 DEBUG:teuthology.orchestra.console.smithi159:power off output: Chassis Power Control: Down/Off
2023-09-21T11:09:15.924 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:09:15.947 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is on
2023-09-21T11:09:18.606 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:09:19.949 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:09:20.060 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is off
2023-09-21T11:09:20.161 INFO:teuthology.orchestra.console.smithi159:Power off completed
2023-09-21T11:09:20.318 INFO:teuthology.lock.ops:unlocked: smithi159.front.sepia.ceph.com
2023-09-21T11:09:21.658 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159
2023-09-21T11:09:27.663 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:09:30.718 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159
2023-09-21T11:09:35.932 DEBUG:teuthology.orchestra.console.smithi121:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T11:09:35.933 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T11:09:36.720 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:09:39.802 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159
2023-09-21T11:09:45.806 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:09:48.862 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159
2023-09-21T11:09:54.962 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:09:56.004 DEBUG:teuthology.orchestra.console.smithi121:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T11:09:56.005 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T11:09:58.042 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159
2023-09-21T11:10:04.048 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:10:07.134 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159
2023-09-21T11:10:13.138 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:10:16.076 DEBUG:teuthology.orchestra.console.smithi121:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T11:10:16.076 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:10:16.219 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159
2023-09-21T11:10:17.085 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:10:21.087 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:10:22.196 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:10:22.221 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:10:25.274 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159
2023-09-21T11:10:26.199 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:10:27.308 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:10:31.279 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:10:31.310 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:10:32.419 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:10:34.334 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159
2023-09-21T11:10:36.422 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:10:37.530 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:10:40.339 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:10:41.533 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:10:42.644 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:10:43.422 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159
2023-09-21T11:10:46.648 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:10:47.757 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:10:49.425 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:10:51.760 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:10:52.506 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159
2023-09-21T11:10:52.869 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:10:56.874 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:10:57.982 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:10:58.509 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:11:01.984 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:11:03.093 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:11:04.606 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159
2023-09-21T11:11:07.094 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:11:08.205 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:11:10.610 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:11:12.209 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:11:13.318 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:11:13.694 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159
2023-09-21T11:11:17.324 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:11:18.434 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:11:19.698 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:11:20.008 WARNING:teuthology.provision.fog.smithi159:[Errno None] Unable to connect to port 22 on 172.21.15.159
2023-09-21T11:11:22.438 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:11:23.547 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:11:26.012 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:11:26.137 ERROR:teuthology.orchestra.connection:Error authenticating with smithi159.front.sepia.ceph.com: Authentication failed.
2023-09-21T11:11:26.142 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-09-21T11:11:26.142 DEBUG:teuthology.orchestra.run.smithi159:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-09-21T11:11:26.145 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T11:11:27.153 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T11:11:27.549 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:11:28.160 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T11:11:28.658 DEBUG:teuthology.orchestra.console.smithi121:check power output: 
2023-09-21T11:11:28.659 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T11:11:28.659 ERROR:teuthology.orchestra.console.smithi121:Failed to power off
2023-09-21T11:11:28.824 INFO:teuthology.lock.ops:unlocked: smithi121.front.sepia.ceph.com
2023-09-21T11:11:29.091 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi121