2023-09-21T11:40:42.198 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-21T11:40:42.199 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-09-21T11:40:42.333 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f93c5d86d30>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f93c5b2eca0>, signals=[15])
2023-09-21T11:40:42.335 INFO:teuthology.lock.ops:Start node 'smithi061.front.sepia.ceph.com' reimaging
2023-09-21T11:40:42.335 INFO:teuthology.lock.ops:Updating [smithi061.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T11:40:42.335 INFO:teuthology.lock.ops:Updating smithi061.front.sepia.ceph.com on lock server
2023-09-21T11:40:42.354 INFO:teuthology.lock.ops:Start node 'smithi100.front.sepia.ceph.com' reimaging
2023-09-21T11:40:42.355 INFO:teuthology.lock.ops:Updating [smithi100.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T11:40:42.355 INFO:teuthology.lock.ops:Updating smithi100.front.sepia.ceph.com on lock server
2023-09-21T11:40:42.356 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /host {"name": "smithi061"}
2023-09-21T11:40:42.379 DEBUG:teuthology.provision.fog.smithi100:FOG request: GET /host {"name": "smithi100"}
2023-09-21T11:40:42.451 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /image {"name": "smithi_ubuntu_22.04"}
2023-09-21T11:40:42.466 DEBUG:teuthology.provision.fog.smithi100:FOG request: GET /image {"name": "smithi_ubuntu_22.04"}
2023-09-21T11:40:42.493 DEBUG:teuthology.provision.fog.smithi061:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-09-21T11:40:42.493 DEBUG:teuthology.provision.fog.smithi061:FOG request: PUT /host/166 {"imageID": 6}
2023-09-21T11:40:42.506 DEBUG:teuthology.provision.fog.smithi100:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-09-21T11:40:42.507 DEBUG:teuthology.provision.fog.smithi100:FOG request: PUT /host/205 {"imageID": 6}
2023-09-21T11:40:42.556 INFO:teuthology.provision.fog.smithi061:Scheduling deploy of ubuntu 22.04
2023-09-21T11:40:42.556 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None
2023-09-21T11:40:42.560 INFO:teuthology.provision.fog.smithi100:Scheduling deploy of ubuntu 22.04
2023-09-21T11:40:42.560 DEBUG:teuthology.provision.fog.smithi100:FOG request: GET /task/active None
2023-09-21T11:40:42.630 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T11:40:42.636 DEBUG:teuthology.provision.fog.smithi100:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T11:40:42.666 DEBUG:teuthology.provision.fog.smithi100:FOG request: POST /host/205/task {"taskTypeID": 1}
2023-09-21T11:40:42.669 DEBUG:teuthology.provision.fog.smithi061:FOG request: POST /host/166/task {"taskTypeID": 1}
2023-09-21T11:40:42.729 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None
2023-09-21T11:40:42.732 DEBUG:teuthology.provision.fog.smithi100:FOG request: GET /task/active None
2023-09-21T11:40:42.839 INFO:teuthology.orchestra.console.smithi100:Power off
2023-09-21T11:40:42.840 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T11:40:42.851 INFO:teuthology.orchestra.console.smithi061:Power off
2023-09-21T11:40:42.852 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T11:40:42.874 DEBUG:teuthology.orchestra.console.smithi061:power off output: Chassis Power Control: Down/Off
2023-09-21T11:40:42.875 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:40:42.898 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is on
2023-09-21T11:40:46.902 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:40:47.010 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is on
2023-09-21T11:40:51.014 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:40:51.123 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is on
2023-09-21T11:40:55.126 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:40:55.234 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is off
2023-09-21T11:40:55.335 INFO:teuthology.orchestra.console.smithi061:Power off completed
2023-09-21T11:40:55.437 INFO:teuthology.orchestra.console.smithi061:Power on
2023-09-21T11:40:55.437 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T11:40:55.458 DEBUG:teuthology.orchestra.console.smithi061:power on output: Chassis Power Control: Up/On
2023-09-21T11:40:55.459 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:40:55.479 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is off
2023-09-21T11:40:59.482 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:40:59.590 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is off
2023-09-21T11:41:02.907 DEBUG:teuthology.orchestra.console.smithi100:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T11:41:02.908 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T11:41:03.592 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:41:03.700 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is on
2023-09-21T11:41:03.801 INFO:teuthology.orchestra.console.smithi061:Power on completed
2023-09-21T11:41:03.902 INFO:teuthology.provision.fog.smithi061:Waiting for deploy to finish
2023-09-21T11:41:03.903 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None
2023-09-21T11:41:18.986 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None
2023-09-21T11:41:22.976 DEBUG:teuthology.orchestra.console.smithi100:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T11:41:22.976 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T11:41:34.070 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None
2023-09-21T11:41:43.048 DEBUG:teuthology.orchestra.console.smithi100:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T11:41:43.048 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:41:44.056 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:41:48.061 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:41:49.143 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None
2023-09-21T11:41:49.170 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:41:53.175 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:41:54.283 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:41:58.286 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:41:59.395 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:42:03.399 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:42:04.228 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None
2023-09-21T11:42:04.507 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:42:08.510 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:42:09.618 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:42:13.622 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:42:14.730 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:42:18.735 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:42:19.312 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None
2023-09-21T11:42:19.842 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:42:23.846 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:42:24.954 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:42:28.959 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:42:30.067 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:42:34.070 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:42:34.395 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None
2023-09-21T11:42:35.178 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:42:39.181 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:42:40.290 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:42:44.295 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:42:45.403 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:42:49.406 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:42:49.463 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None
2023-09-21T11:42:50.514 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:42:54.518 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:42:55.626 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:42:55.627 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T11:42:55.628 ERROR:teuthology.orchestra.console.smithi100:Failed to power off
2023-09-21T11:42:55.628 INFO:teuthology.orchestra.console.smithi100:Power on
2023-09-21T11:42:55.629 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T11:43:04.546 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None
2023-09-21T11:43:15.695 DEBUG:teuthology.orchestra.console.smithi100:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T11:43:15.696 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T11:43:19.621 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None
2023-09-21T11:43:34.706 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None
2023-09-21T11:43:35.764 DEBUG:teuthology.orchestra.console.smithi100:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T11:43:35.764 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T11:43:49.791 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None
2023-09-21T11:43:55.832 DEBUG:teuthology.orchestra.console.smithi100:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T11:43:55.832 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:43:56.842 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:44:00.846 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:44:01.954 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:44:04.870 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None
2023-09-21T11:44:05.957 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:44:07.065 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:44:11.067 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:44:12.175 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:44:16.180 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:44:17.289 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:44:19.958 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None
2023-09-21T11:44:20.031 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:44:21.292 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:44:22.401 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:44:26.406 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:44:27.517 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:44:31.518 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:44:32.626 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:44:36.630 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:44:37.742 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:44:38.554 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61
2023-09-21T11:44:41.746 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:44:42.854 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:44:44.558 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:44:46.857 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:44:47.614 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61
2023-09-21T11:44:47.966 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:44:51.970 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:44:53.078 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:44:53.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:44:56.698 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61
2023-09-21T11:44:57.079 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:44:58.188 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:45:02.190 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:45:02.700 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:45:03.298 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:45:05.754 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61
2023-09-21T11:45:07.300 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:45:08.409 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:45:08.410 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T11:45:08.410 DEBUG:teuthology.provision.fog.smithi100:Canceling deploy task with ID 499360
2023-09-21T11:45:08.410 DEBUG:teuthology.provision.fog.smithi100:FOG request: DELETE /task/cancel {"id": 499360}
2023-09-21T11:45:08.474 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi100
2023-09-21T11:45:08.475 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-09-21T11:45:08.604 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=b4be7fb77b1d4d6a9ea229b89ee6e6e4
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi100
2023-09-21T11:45:08.607 INFO:teuthology.nuke:Checking targets against current locks
2023-09-21T11:45:08.674 DEBUG:teuthology.nuke:shortname: smithi061
2023-09-21T11:45:08.675 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T11:45:08.697 DEBUG:teuthology.nuke:shortname: smithi100
2023-09-21T11:45:08.697 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T11:45:08.701 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi061.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-09-20_16:32:45-fs-wip-lusov-config-format-distro-default-smithi/7399806', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 11:40:41.322889', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPC/TwRaoYtpG9v6trZPpChAhjZOZO/ORq3nS1ekVDtWf/aj/So3+3Kh1ak/VeMXCexT1mFN4nwc5TFPbdzVvFE='}
2023-09-21T11:45:08.723 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi100.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-09-20_16:32:45-fs-wip-lusov-config-format-distro-default-smithi/7399806', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 11:40:41.321356', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDWK+DDsnn36F3dyE7RoAnqrhNL1tuuGKQNOB7wB5pQmZCY+ochzcrVfhhHITZeFdeJ0if7J71l46+Pfb1V4URk='}
2023-09-21T11:45:08.758 INFO:teuthology.orchestra.console.smithi061:Power off
2023-09-21T11:45:08.759 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T11:45:08.780 DEBUG:teuthology.orchestra.console.smithi061:power off output: Chassis Power Control: Down/Off
2023-09-21T11:45:08.780 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:45:08.795 INFO:teuthology.orchestra.console.smithi100:Power off
2023-09-21T11:45:08.796 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T11:45:08.805 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is on
2023-09-21T11:45:11.758 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:45:12.807 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:45:12.917 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is off
2023-09-21T11:45:13.019 INFO:teuthology.orchestra.console.smithi061:Power off completed
2023-09-21T11:45:13.171 INFO:teuthology.lock.ops:unlocked: smithi061.front.sepia.ceph.com
2023-09-21T11:45:14.810 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61
2023-09-21T11:45:20.817 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:45:23.898 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61
2023-09-21T11:45:28.864 DEBUG:teuthology.orchestra.console.smithi100:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T11:45:28.864 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T11:45:29.901 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:45:32.954 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61
2023-09-21T11:45:38.961 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:45:42.042 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61
2023-09-21T11:45:48.046 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:45:48.940 DEBUG:teuthology.orchestra.console.smithi100:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T11:45:48.940 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T11:45:51.098 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61
2023-09-21T11:45:57.102 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:46:00.154 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61
2023-09-21T11:46:06.160 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:46:09.007 DEBUG:teuthology.orchestra.console.smithi100:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T11:46:09.008 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:46:09.246 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61
2023-09-21T11:46:10.017 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:46:14.019 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:46:15.127 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:46:15.249 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:46:18.330 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61
2023-09-21T11:46:19.130 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:46:20.239 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:46:24.244 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:46:24.333 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:46:25.354 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:46:27.390 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61
2023-09-21T11:46:29.357 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:46:30.466 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:46:33.395 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:46:34.469 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:46:35.578 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:46:36.474 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61
2023-09-21T11:46:39.583 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:46:40.691 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:46:42.478 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:46:44.695 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:46:45.530 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61
2023-09-21T11:46:45.806 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:46:49.810 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:46:50.921 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:46:51.533 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:46:54.586 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61
2023-09-21T11:46:54.923 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:46:56.032 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:47:00.037 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:47:00.589 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:47:01.147 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:47:03.642 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61
2023-09-21T11:47:05.149 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:47:06.259 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:47:09.647 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:47:10.262 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:47:11.373 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:47:15.375 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:47:16.485 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:47:20.490 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:47:21.599 DEBUG:teuthology.orchestra.console.smithi100:check power output: 
2023-09-21T11:47:21.600 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T11:47:21.600 ERROR:teuthology.orchestra.console.smithi100:Failed to power off
2023-09-21T11:47:21.754 INFO:teuthology.lock.ops:unlocked: smithi100.front.sepia.ceph.com
2023-09-21T11:47:21.996 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi100