2024-01-17T20:07:14.644 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-01-17T20:07:14.645 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-17T20:07:14.778 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f9f7db93f40>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f9f7ebf9c10>, signals=[15])
2024-01-17T20:07:14.779 INFO:teuthology.lock.ops:Start node 'smithi029.front.sepia.ceph.com' reimaging
2024-01-17T20:07:14.779 INFO:teuthology.lock.ops:Updating [smithi029.front.sepia.ceph.com]: reset os type and version on server
2024-01-17T20:07:14.779 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server
2024-01-17T20:07:14.832 INFO:teuthology.lock.ops:Start node 'smithi061.front.sepia.ceph.com' reimaging
2024-01-17T20:07:14.833 INFO:teuthology.lock.ops:Updating [smithi061.front.sepia.ceph.com]: reset os type and version on server
2024-01-17T20:07:14.833 INFO:teuthology.lock.ops:Updating smithi061.front.sepia.ceph.com on lock server
2024-01-17T20:07:14.943 DEBUG:teuthology.provision.fog.smithi029:Requesting image smithi_centos_9.stream (ID 10)
2024-01-17T20:07:14.975 DEBUG:teuthology.provision.fog.smithi061:Requesting image smithi_centos_9.stream (ID 10)
2024-01-17T20:07:14.983 INFO:teuthology.provision.fog.smithi029:Scheduling deploy of centos 9.stream
2024-01-17T20:07:15.013 INFO:teuthology.provision.fog.smithi061:Scheduling deploy of centos 9.stream
2024-01-17T20:07:15.295 INFO:teuthology.orchestra.console.smithi029:Power off
2024-01-17T20:07:15.296 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-01-17T20:07:15.307 INFO:teuthology.orchestra.console.smithi061:Power off
2024-01-17T20:07:15.307 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-01-17T20:07:15.319 DEBUG:teuthology.orchestra.console.smithi029:power off output: Chassis Power Control: Down/Off
2024-01-17T20:07:15.320 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:07:15.341 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is off
2024-01-17T20:07:15.441 INFO:teuthology.orchestra.console.smithi029:Power off completed
2024-01-17T20:07:15.542 INFO:teuthology.orchestra.console.smithi029:Power on
2024-01-17T20:07:15.542 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-01-17T20:07:15.563 DEBUG:teuthology.orchestra.console.smithi029:power on output: Chassis Power Control: Up/On
2024-01-17T20:07:15.563 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:07:15.584 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is off
2024-01-17T20:07:19.585 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:07:19.694 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is off
2024-01-17T20:07:23.698 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:07:23.807 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on
2024-01-17T20:07:23.907 INFO:teuthology.orchestra.console.smithi029:Power on completed
2024-01-17T20:07:24.008 INFO:teuthology.provision.fog.smithi029:Waiting for deploy to finish
2024-01-17T20:07:35.383 DEBUG:teuthology.orchestra.console.smithi061:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2024-01-17T20:07:35.383 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-01-17T20:07:55.454 DEBUG:teuthology.orchestra.console.smithi061:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2024-01-17T20:07:55.455 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-01-17T20:08:15.527 DEBUG:teuthology.orchestra.console.smithi061:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2024-01-17T20:08:15.527 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:08:16.536 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:08:20.540 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:08:21.649 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:08:25.651 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:08:26.760 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:08:30.764 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:08:31.873 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:08:35.878 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:08:36.987 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:08:40.990 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:08:42.099 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:08:46.103 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:08:47.212 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:08:51.214 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:08:52.323 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:08:56.326 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:08:57.435 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:09:01.436 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:09:02.544 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:09:06.549 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:09:07.658 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:09:11.660 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:09:12.768 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:09:16.772 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:09:17.880 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:09:21.882 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:09:22.991 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:09:26.995 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:09:28.104 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:09:28.105 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2024-01-17T20:09:28.105 ERROR:teuthology.orchestra.console.smithi061:Failed to power off
2024-01-17T20:09:28.105 INFO:teuthology.orchestra.console.smithi061:Power on
2024-01-17T20:09:28.105 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-01-17T20:09:48.174 DEBUG:teuthology.orchestra.console.smithi061:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2024-01-17T20:09:48.174 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-01-17T20:10:08.247 DEBUG:teuthology.orchestra.console.smithi061:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2024-01-17T20:10:08.247 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-01-17T20:10:24.759 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-17T20:10:28.319 DEBUG:teuthology.orchestra.console.smithi061:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2024-01-17T20:10:28.319 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:10:29.327 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:10:33.332 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:10:34.440 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:10:38.444 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:10:39.552 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:10:43.132 WARNING:teuthology.provision.fog.smithi029:[Errno None] Unable to connect to port 22 on 172.21.15.29
2024-01-17T20:10:43.553 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:10:44.662 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:10:48.664 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:10:49.135 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-17T20:10:49.772 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:10:52.188 WARNING:teuthology.provision.fog.smithi029:[Errno None] Unable to connect to port 22 on 172.21.15.29
2024-01-17T20:10:53.774 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:10:54.882 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:10:58.193 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-17T20:10:58.884 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:10:59.993 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:11:01.244 WARNING:teuthology.provision.fog.smithi029:[Errno None] Unable to connect to port 22 on 172.21.15.29
2024-01-17T20:11:03.996 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:11:05.106 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:11:07.248 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-17T20:11:09.109 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:11:10.218 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:11:10.300 WARNING:teuthology.provision.fog.smithi029:[Errno None] Unable to connect to port 22 on 172.21.15.29
2024-01-17T20:11:14.223 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:11:15.334 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:11:16.303 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-17T20:11:19.338 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:11:20.446 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:11:24.448 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:11:25.557 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:11:29.561 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:11:30.670 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:11:34.672 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:11:35.782 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:11:39.783 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:11:40.895 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:11:40.896 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2024-01-17T20:11:40.896 DEBUG:teuthology.provision.fog.smithi061:Canceling deploy task with ID 719542
2024-01-17T20:11:41.041 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi061
2024-01-17T20:11:41.043 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-17T20:11:41.337 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=55869d57d3c4478586f247cbe267d090
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi061
2024-01-17T20:11:41.339 INFO:teuthology.nuke:Checking targets against current locks
2024-01-17T20:11:41.420 DEBUG:teuthology.nuke:shortname: smithi029
2024-01-17T20:11:41.420 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-01-17T20:11:41.444 DEBUG:teuthology.nuke:shortname: smithi061
2024-01-17T20:11:41.444 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-01-17T20:11:41.447 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amaredia-2024-01-17_20:02:54-rgw:singleton-wip-rgw-add-cap-user-without-keys-distro-default-smithi/7519193', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-17 20:07:13.661070', 'locked_by': 'scheduled_amaredia@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJYIdAtf0hC1h9XNFKhBOKVdEiQJf0jpwrIVgz6Ur+UKI7wahtTYv4ldlxYRYcrzHjS6BeGEqYFKPawh9clE/jQ='}
2024-01-17T20:11:41.472 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi061.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amaredia-2024-01-17_20:02:54-rgw:singleton-wip-rgw-add-cap-user-without-keys-distro-default-smithi/7519193', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-17 20:07:13.662881', 'locked_by': 'scheduled_amaredia@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG8NuJB1fW7JNXTUxAOA5cZYhMmvcAfm0JPn1vogOrF0hTNSvcXWbbDW+EIcD9tK8JQl4YgLN4huMMwZ60njGT0='}
2024-01-17T20:11:41.500 INFO:teuthology.orchestra.console.smithi029:Power off
2024-01-17T20:11:41.500 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-01-17T20:11:41.527 DEBUG:teuthology.orchestra.console.smithi029:power off output: Chassis Power Control: Down/Off
2024-01-17T20:11:41.528 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:11:41.542 INFO:teuthology.orchestra.console.smithi061:Power off
2024-01-17T20:11:41.543 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-01-17T20:11:41.552 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on
2024-01-17T20:11:45.556 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:11:45.665 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on
2024-01-17T20:11:49.669 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:11:49.779 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on
2024-01-17T20:11:53.784 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:11:53.893 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is off
2024-01-17T20:11:53.994 INFO:teuthology.orchestra.console.smithi029:Power off completed
2024-01-17T20:11:54.151 INFO:teuthology.lock.ops:unlocked: smithi029.front.sepia.ceph.com
2024-01-17T20:12:01.618 DEBUG:teuthology.orchestra.console.smithi061:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2024-01-17T20:12:01.618 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-01-17T20:12:16.320 WARNING:teuthology.provision.fog.smithi029:timed out
2024-01-17T20:12:21.692 DEBUG:teuthology.orchestra.console.smithi061:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2024-01-17T20:12:21.692 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-01-17T20:12:22.323 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-17T20:12:40.893 WARNING:teuthology.provision.fog.smithi029:[Errno None] Unable to connect to port 22 on 172.21.15.29
2024-01-17T20:12:41.769 DEBUG:teuthology.orchestra.console.smithi061:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2024-01-17T20:12:41.769 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:12:42.780 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:12:46.781 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:12:46.895 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-17T20:12:47.893 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:12:49.952 WARNING:teuthology.provision.fog.smithi029:[Errno None] Unable to connect to port 22 on 172.21.15.29
2024-01-17T20:12:51.896 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:12:53.005 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:12:55.956 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-17T20:12:57.007 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:12:58.116 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:12:59.036 WARNING:teuthology.provision.fog.smithi029:[Errno None] Unable to connect to port 22 on 172.21.15.29
2024-01-17T20:13:02.120 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:13:03.230 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:13:05.040 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-17T20:13:07.232 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:13:08.092 WARNING:teuthology.provision.fog.smithi029:[Errno None] Unable to connect to port 22 on 172.21.15.29
2024-01-17T20:13:08.341 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:13:12.343 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:13:13.453 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:13:14.095 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-17T20:13:17.148 WARNING:teuthology.provision.fog.smithi029:[Errno None] Unable to connect to port 22 on 172.21.15.29
2024-01-17T20:13:17.455 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:13:18.564 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:13:22.568 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:13:23.150 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-17T20:13:23.678 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:13:26.204 WARNING:teuthology.provision.fog.smithi029:[Errno None] Unable to connect to port 22 on 172.21.15.29
2024-01-17T20:13:27.680 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:13:28.790 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:13:32.209 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-17T20:13:32.791 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:13:33.902 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:13:35.260 WARNING:teuthology.provision.fog.smithi029:[Errno None] Unable to connect to port 22 on 172.21.15.29
2024-01-17T20:13:37.905 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:13:39.015 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:13:41.264 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-17T20:13:43.018 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:13:44.128 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:13:44.316 WARNING:teuthology.provision.fog.smithi029:[Errno None] Unable to connect to port 22 on 172.21.15.29
2024-01-17T20:13:48.132 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:13:49.240 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:13:50.319 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-17T20:13:53.244 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-17T20:13:53.372 WARNING:teuthology.provision.fog.smithi029:[Errno None] Unable to connect to port 22 on 172.21.15.29
2024-01-17T20:13:54.354 DEBUG:teuthology.orchestra.console.smithi061:check power output: 
2024-01-17T20:13:54.354 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2024-01-17T20:13:54.354 ERROR:teuthology.orchestra.console.smithi061:Failed to power off
2024-01-17T20:13:54.515 INFO:teuthology.lock.ops:unlocked: smithi061.front.sepia.ceph.com
2024-01-17T20:13:54.792 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi061