2023-11-04T12:27:44.344 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-04T12:27:44.345 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-04T12:27:44.452 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2adc5b4670>, signals=[15]) 2023-11-04T12:27:44.454 INFO:teuthology.lock.ops:Start node 'smithi110.front.sepia.ceph.com' reimaging 2023-11-04T12:27:44.454 INFO:teuthology.lock.ops:Updating [smithi110.front.sepia.ceph.com]: reset os type and version on server 2023-11-04T12:27:44.455 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2023-11-04T12:27:44.483 DEBUG:teuthology.provision.fog.smithi110:FOG request: GET /host {"name": "smithi110"} 2023-11-04T12:27:44.563 DEBUG:teuthology.provision.fog.smithi110:FOG request: GET /image {"name": "smithi_centos_9.stream"} 2023-11-04T12:27:44.613 DEBUG:teuthology.provision.fog.smithi110:Requesting image smithi_centos_9.stream (ID 10) 2023-11-04T12:27:44.613 DEBUG:teuthology.provision.fog.smithi110:FOG request: PUT /host/215 {"imageID": 10} 2023-11-04T12:27:44.665 INFO:teuthology.provision.fog.smithi110:Scheduling deploy of centos 9.stream 2023-11-04T12:27:44.666 DEBUG:teuthology.provision.fog.smithi110:FOG request: GET /task/active None 2023-11-04T12:27:44.741 DEBUG:teuthology.provision.fog.smithi110:FOG request: GET /tasktype {"name": "deploy"} 2023-11-04T12:27:44.778 DEBUG:teuthology.provision.fog.smithi110:FOG request: POST /host/215/task {"taskTypeID": 1} 2023-11-04T12:27:44.841 DEBUG:teuthology.provision.fog.smithi110:FOG request: GET /task/active None 2023-11-04T12:27:44.949 INFO:teuthology.orchestra.console.smithi110:Power off 2023-11-04T12:27:44.951 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-04T12:28:05.021 DEBUG:teuthology.orchestra.console.smithi110:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-11-04T12:28:05.022 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-04T12:28:25.090 DEBUG:teuthology.orchestra.console.smithi110:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-11-04T12:28:25.091 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-04T12:28:45.161 DEBUG:teuthology.orchestra.console.smithi110:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-11-04T12:28:45.162 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:28:46.172 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:28:50.174 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:28:51.282 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:28:55.286 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:28:56.394 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:29:00.398 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:29:01.506 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:29:05.511 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:29:06.621 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:29:10.622 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:29:11.730 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:29:15.734 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:29:16.843 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:29:20.846 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:29:21.956 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:29:25.958 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:29:27.066 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:29:31.070 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:29:32.180 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:29:36.185 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:29:37.293 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:29:41.298 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:29:42.409 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:29:46.413 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:29:47.522 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:29:51.526 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:29:52.637 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:29:56.642 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:29:57.750 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:29:57.751 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds 2023-11-04T12:29:57.751 ERROR:teuthology.orchestra.console.smithi110:Failed to power off 2023-11-04T12:29:57.752 INFO:teuthology.orchestra.console.smithi110:Power on 2023-11-04T12:29:57.752 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-11-04T12:30:17.821 DEBUG:teuthology.orchestra.console.smithi110:power on output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-11-04T12:30:17.821 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-11-04T12:30:37.893 DEBUG:teuthology.orchestra.console.smithi110:power on output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-11-04T12:30:37.894 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-11-04T12:30:57.966 DEBUG:teuthology.orchestra.console.smithi110:power on output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-11-04T12:30:57.966 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:30:58.974 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:31:02.978 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:31:04.086 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:31:08.090 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:31:09.200 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:31:13.202 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:31:14.310 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:31:18.314 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:31:19.422 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:31:23.426 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:31:24.535 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:31:28.538 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:31:29.648 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:31:33.651 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:31:34.764 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:31:38.769 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:31:39.878 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:31:43.882 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:31:44.991 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:31:48.994 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:31:50.104 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:31:54.106 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:31:55.218 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:31:59.222 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:32:00.332 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:32:04.334 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:32:05.442 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:32:09.446 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:32:10.554 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:32:10.555 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds 2023-11-04T12:32:10.555 DEBUG:teuthology.provision.fog.smithi110:Canceling deploy task with ID 583645 2023-11-04T12:32:10.555 DEBUG:teuthology.provision.fog.smithi110:FOG request: DELETE /task/cancel {"id": 583645} 2023-11-04T12:32:10.612 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi110 2023-11-04T12:32:10.613 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-04T12:32:10.759 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=7a5d6b49233645ae893fe982d554c79e Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi110 2023-11-04T12:32:10.762 INFO:teuthology.nuke:Checking targets against current locks 2023-11-04T12:32:10.825 DEBUG:teuthology.nuke:shortname: smithi110 2023-11-04T12:32:10.826 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-04T12:32:10.849 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-04_03:30:59-rbd-main-distro-default-smithi/7446385', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-04 12:27:43.582748', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD/JYy4AtoB411+YGlWfUP43f/QjbYH8XDANNLXq7IDAFFM4TcAlSi9j1pVtRJtRXIflalipYSTpImX3zs//PXs='} 2023-11-04T12:32:10.897 INFO:teuthology.orchestra.console.smithi110:Power off 2023-11-04T12:32:10.898 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-04T12:32:30.968 DEBUG:teuthology.orchestra.console.smithi110:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-11-04T12:32:30.969 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-04T12:32:51.036 DEBUG:teuthology.orchestra.console.smithi110:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-11-04T12:32:51.037 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-04T12:33:11.104 DEBUG:teuthology.orchestra.console.smithi110:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-11-04T12:33:11.105 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:33:12.114 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:33:16.117 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:33:17.226 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:33:21.231 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:33:22.340 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:33:26.342 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:33:27.451 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:33:31.454 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:33:32.563 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:33:36.567 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:33:37.675 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:33:41.678 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:33:42.787 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:33:46.790 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:33:47.900 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:33:51.905 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:33:53.014 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:33:57.018 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:33:58.128 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:34:02.133 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:34:03.243 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:34:07.246 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:34:08.356 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:34:12.360 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:34:13.470 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:34:17.475 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:34:18.584 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:34:22.589 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:34:23.698 DEBUG:teuthology.orchestra.console.smithi110:check power output: 2023-11-04T12:34:23.699 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds 2023-11-04T12:34:23.699 ERROR:teuthology.orchestra.console.smithi110:Failed to power off 2023-11-04T12:34:23.853 INFO:teuthology.lock.ops:unlocked: smithi110.front.sepia.ceph.com 2023-11-04T12:34:23.959 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 58, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi110