2023-11-04T12:24:22.148 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-04T12:24:22.150 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-04T12:24:22.270 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9767c13670>, signals=[15]) 2023-11-04T12:24:22.271 INFO:teuthology.lock.ops:Start node 'smithi125.front.sepia.ceph.com' reimaging 2023-11-04T12:24:22.271 INFO:teuthology.lock.ops:Updating [smithi125.front.sepia.ceph.com]: reset os type and version on server 2023-11-04T12:24:22.272 INFO:teuthology.lock.ops:Updating smithi125.front.sepia.ceph.com on lock server 2023-11-04T12:24:22.296 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /host {"name": "smithi125"} 2023-11-04T12:24:22.378 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /image {"name": "smithi_ubuntu_22.04"} 2023-11-04T12:24:22.438 DEBUG:teuthology.provision.fog.smithi125:Requesting image smithi_ubuntu_22.04 (ID 6) 2023-11-04T12:24:22.438 DEBUG:teuthology.provision.fog.smithi125:FOG request: PUT /host/230 {"imageID": 6} 2023-11-04T12:24:22.484 INFO:teuthology.provision.fog.smithi125:Scheduling deploy of ubuntu 22.04 2023-11-04T12:24:22.484 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /task/active None 2023-11-04T12:24:22.545 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /tasktype {"name": "deploy"} 2023-11-04T12:24:22.581 DEBUG:teuthology.provision.fog.smithi125:FOG request: POST /host/230/task {"taskTypeID": 1} 2023-11-04T12:24:22.640 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /task/active None 2023-11-04T12:24:22.734 INFO:teuthology.orchestra.console.smithi125:Power off 2023-11-04T12:24:22.735 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-04T12:24:42.805 DEBUG:teuthology.orchestra.console.smithi125:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-11-04T12:24:42.806 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-04T12:25:02.872 DEBUG:teuthology.orchestra.console.smithi125:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-11-04T12:25:02.873 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-04T12:25:22.945 DEBUG:teuthology.orchestra.console.smithi125:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-11-04T12:25:22.945 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:25:23.954 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:25:27.958 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:25:29.068 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:25:33.070 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:25:34.180 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:25:38.182 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:25:39.291 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:25:43.295 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:25:44.403 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:25:48.406 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:25:49.514 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:25:53.518 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:25:54.626 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:25:58.630 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:25:59.738 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:26:03.743 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:26:04.852 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:26:08.854 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:26:09.964 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:26:13.966 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:26:15.075 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:26:19.078 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:26:20.186 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:26:24.190 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:26:25.301 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:26:29.302 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:26:30.410 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:26:34.414 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:26:35.522 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:26:35.523 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds 2023-11-04T12:26:35.524 ERROR:teuthology.orchestra.console.smithi125:Failed to power off 2023-11-04T12:26:35.524 INFO:teuthology.orchestra.console.smithi125:Power on 2023-11-04T12:26:35.524 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-11-04T12:26:55.595 DEBUG:teuthology.orchestra.console.smithi125:power on output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-11-04T12:26:55.596 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-11-04T12:27:15.665 DEBUG:teuthology.orchestra.console.smithi125:power on output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-11-04T12:27:15.666 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-11-04T12:27:35.736 DEBUG:teuthology.orchestra.console.smithi125:power on output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-11-04T12:27:35.736 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:27:36.745 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:27:40.750 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:27:41.858 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:27:45.862 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:27:46.970 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:27:50.973 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:27:52.082 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:27:56.086 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:27:57.194 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:28:01.198 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:28:02.306 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:28:06.309 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:28:07.418 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:28:11.422 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:28:12.530 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:28:16.534 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:28:17.643 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:28:21.646 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:28:22.755 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:28:26.758 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:28:27.866 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:28:31.870 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:28:32.978 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:28:36.982 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:28:38.090 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:28:42.094 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:28:43.203 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:28:47.206 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:28:48.314 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:28:48.315 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds 2023-11-04T12:28:48.315 DEBUG:teuthology.provision.fog.smithi125:Canceling deploy task with ID 583635 2023-11-04T12:28:48.316 DEBUG:teuthology.provision.fog.smithi125:FOG request: DELETE /task/cancel {"id": 583635} 2023-11-04T12:28:48.403 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi125 2023-11-04T12:28:48.404 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-04T12:28:48.552 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=6dbeeeb58d494020b993b0513848073a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi125 2023-11-04T12:28:48.555 INFO:teuthology.nuke:Checking targets against current locks 2023-11-04T12:28:48.614 DEBUG:teuthology.nuke:shortname: smithi125 2023-11-04T12:28:48.615 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-04T12:28:48.638 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi125.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-04_03:30:59-rados-main-distro-default-smithi/7446380', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-04 12:24:21.307035', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNbt3MkpAZU6m5fLvPbrEcWgxMPqXIsnqw2IBIEA8O8PMPSV4xy/VUJp6m41SMxMBR27TLCMk1iYMJ1+++W7lNw='} 2023-11-04T12:28:48.691 INFO:teuthology.orchestra.console.smithi125:Power off 2023-11-04T12:28:48.692 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-04T12:29:08.762 DEBUG:teuthology.orchestra.console.smithi125:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-11-04T12:29:08.764 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-04T12:29:28.836 DEBUG:teuthology.orchestra.console.smithi125:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-11-04T12:29:28.837 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-04T12:29:48.908 DEBUG:teuthology.orchestra.console.smithi125:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-11-04T12:29:48.909 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:29:49.918 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:29:53.919 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:29:55.032 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:29:59.034 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:30:00.144 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:30:04.146 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:30:05.256 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:30:09.259 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:30:10.367 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:30:14.370 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:30:15.480 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:30:19.482 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:30:20.592 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:30:24.595 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:30:25.704 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:30:29.706 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:30:30.814 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:30:34.818 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:30:35.927 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:30:39.930 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:30:41.038 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:30:45.042 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:30:46.151 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:30:50.154 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:30:51.264 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:30:55.266 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:30:56.376 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:31:00.378 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-04T12:31:01.487 DEBUG:teuthology.orchestra.console.smithi125:check power output: 2023-11-04T12:31:01.487 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds 2023-11-04T12:31:01.488 ERROR:teuthology.orchestra.console.smithi125:Failed to power off 2023-11-04T12:31:01.650 INFO:teuthology.lock.ops:unlocked: smithi125.front.sepia.ceph.com 2023-11-04T12:31:01.765 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 58, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi125