2024-04-10T00:29:47.379 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-10T00:29:47.380 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-10T00:29:47.569 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f936753dc10>, signals=[15]) 2024-04-10T00:29:47.569 INFO:teuthology.lock.ops:Start node 'smithi037.front.sepia.ceph.com' reimaging 2024-04-10T00:29:47.570 INFO:teuthology.lock.ops:Updating [smithi037.front.sepia.ceph.com]: reset os type and version on server 2024-04-10T00:29:47.570 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server 2024-04-10T00:29:47.591 INFO:teuthology.lock.ops:Start node 'smithi045.front.sepia.ceph.com' reimaging 2024-04-10T00:29:47.591 INFO:teuthology.lock.ops:Updating [smithi045.front.sepia.ceph.com]: reset os type and version on server 2024-04-10T00:29:47.591 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server 2024-04-10T00:29:47.607 INFO:teuthology.lock.ops:Start node 'smithi081.front.sepia.ceph.com' reimaging 2024-04-10T00:29:47.607 INFO:teuthology.lock.ops:Updating [smithi081.front.sepia.ceph.com]: reset os type and version on server 2024-04-10T00:29:47.607 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2024-04-10T00:29:47.697 DEBUG:teuthology.provision.fog.smithi037:Requesting image smithi_centos_8.stream (ID 3) 2024-04-10T00:29:47.717 DEBUG:teuthology.provision.fog.smithi045:Requesting image smithi_centos_8.stream (ID 3) 2024-04-10T00:29:47.723 DEBUG:teuthology.provision.fog.smithi081:Requesting image smithi_centos_8.stream (ID 3) 2024-04-10T00:29:47.742 INFO:teuthology.provision.fog.smithi037:Scheduling deploy of centos 8.stream 2024-04-10T00:29:47.763 INFO:teuthology.provision.fog.smithi045:Scheduling deploy of centos 8.stream 2024-04-10T00:29:47.766 INFO:teuthology.provision.fog.smithi081:Scheduling deploy of centos 8.stream 2024-04-10T00:29:47.979 INFO:teuthology.orchestra.console.smithi037:Power off 2024-04-10T00:29:47.980 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T00:29:47.997 INFO:teuthology.orchestra.console.smithi081:Power off 2024-04-10T00:29:47.997 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T00:29:48.014 INFO:teuthology.orchestra.console.smithi045:Power off 2024-04-10T00:29:48.014 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T00:29:48.021 DEBUG:teuthology.orchestra.console.smithi037:power off output: Chassis Power Control: Down/Off 2024-04-10T00:29:48.021 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:29:48.029 DEBUG:teuthology.orchestra.console.smithi081:power off output: Chassis Power Control: Down/Off 2024-04-10T00:29:48.030 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:29:48.039 DEBUG:teuthology.orchestra.console.smithi045:power off output: Chassis Power Control: Down/Off 2024-04-10T00:29:48.039 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:29:48.050 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is on 2024-04-10T00:29:48.055 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on 2024-04-10T00:29:48.060 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off 2024-04-10T00:29:48.161 INFO:teuthology.orchestra.console.smithi045:Power off completed 2024-04-10T00:29:48.262 INFO:teuthology.orchestra.console.smithi045:Power on 2024-04-10T00:29:48.262 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-04-10T00:29:48.286 DEBUG:teuthology.orchestra.console.smithi045:power on output: Chassis Power Control: Up/On 2024-04-10T00:29:48.286 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:29:48.310 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off 2024-04-10T00:29:52.054 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:29:52.062 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:29:52.164 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off 2024-04-10T00:29:52.169 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is off 2024-04-10T00:29:52.266 INFO:teuthology.orchestra.console.smithi037:Power off completed 2024-04-10T00:29:52.270 INFO:teuthology.orchestra.console.smithi081:Power off completed 2024-04-10T00:29:52.311 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:29:52.367 INFO:teuthology.orchestra.console.smithi037:Power on 2024-04-10T00:29:52.367 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-04-10T00:29:52.374 INFO:teuthology.orchestra.console.smithi081:Power on 2024-04-10T00:29:52.374 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-04-10T00:29:52.388 DEBUG:teuthology.orchestra.console.smithi037:power on output: Chassis Power Control: Up/On 2024-04-10T00:29:52.389 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:29:52.398 DEBUG:teuthology.orchestra.console.smithi081:power on output: Chassis Power Control: Up/On 2024-04-10T00:29:52.398 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:29:52.411 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off 2024-04-10T00:29:52.419 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is off 2024-04-10T00:29:52.420 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on 2024-04-10T00:29:52.520 INFO:teuthology.orchestra.console.smithi045:Power on completed 2024-04-10T00:29:52.621 INFO:teuthology.provision.fog.smithi045:Waiting for deploy to finish 2024-04-10T00:29:56.414 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:29:56.421 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:29:56.524 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off 2024-04-10T00:29:56.531 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on 2024-04-10T00:29:56.632 INFO:teuthology.orchestra.console.smithi081:Power on completed 2024-04-10T00:29:56.732 INFO:teuthology.provision.fog.smithi081:Waiting for deploy to finish 2024-04-10T00:30:00.529 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:30:00.639 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off 2024-04-10T00:30:04.642 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:30:04.751 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off 2024-04-10T00:30:08.753 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:30:08.862 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off 2024-04-10T00:30:12.864 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:30:12.973 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off 2024-04-10T00:30:16.978 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:30:17.087 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off 2024-04-10T00:30:21.092 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:30:21.200 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off 2024-04-10T00:30:25.202 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:30:25.311 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off 2024-04-10T00:30:29.314 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:30:29.424 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off 2024-04-10T00:30:33.426 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:30:33.535 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off 2024-04-10T00:30:37.539 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:30:37.648 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off 2024-04-10T00:30:41.650 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:30:41.760 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off 2024-04-10T00:30:45.763 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:30:45.873 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off 2024-04-10T00:30:49.878 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:30:49.987 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off 2024-04-10T00:30:49.988 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds 2024-04-10T00:30:50.088 DEBUG:teuthology.provision.fog.smithi037:Canceling deploy task with ID 942056 2024-04-10T00:30:50.246 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi037 2024-04-10T00:30:50.248 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-10T00:30:50.569 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=781369d81b544f5abbac0175d7429dad Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi037 2024-04-10T00:30:50.571 INFO:teuthology.nuke:Checking targets against current locks 2024-04-10T00:30:50.635 DEBUG:teuthology.nuke:shortname: smithi037 2024-04-10T00:30:50.635 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T00:30:50.659 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi037.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-09_14:35:50-rados-wip-yuri5-testing-2024-03-21-0833-distro-default-smithi/7648751', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 00:29:46.311005', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIl0Y/Jmjrcvd1iiO7UAsGGgHoEpb56A5F3PvB9K8fcKyT64vwJTnlRoDDOPUt8EYnNsSte7G3nQW5M7+6vnUCs='} 2024-04-10T00:30:50.662 DEBUG:teuthology.nuke:shortname: smithi045 2024-04-10T00:30:50.662 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T00:30:50.678 DEBUG:teuthology.nuke:shortname: smithi081 2024-04-10T00:30:50.679 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T00:30:50.686 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-09_14:35:50-rados-wip-yuri5-testing-2024-03-21-0833-distro-default-smithi/7648751', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 00:29:46.314659', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBxxC4mXGIVHCURyV8+qn6Rl9D/BOKlr/GoTZsX84uwN0O/idVTEwOvofUq2j9iQ6NrJYE34pjcLqJu3G/6FGIs='} 2024-04-10T00:30:50.695 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-09_14:35:50-rados-wip-yuri5-testing-2024-03-21-0833-distro-default-smithi/7648751', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 00:29:46.312918', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOqJ4POO5Pmo7xoEApdwPlvKd7qLEnJH5RMWk8I9k0u6fnnKN4SBPI2g3rWXc6q6YYrQtrVI5K2TatR/zROFBew='} 2024-04-10T00:30:50.712 INFO:teuthology.orchestra.console.smithi037:Power off 2024-04-10T00:30:50.712 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T00:30:50.733 INFO:teuthology.orchestra.console.smithi045:Power off 2024-04-10T00:30:50.734 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T00:30:50.747 DEBUG:teuthology.orchestra.console.smithi037:power off output: Chassis Power Control: Down/Off 2024-04-10T00:30:50.747 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:30:50.756 DEBUG:teuthology.orchestra.console.smithi045:power off output: Chassis Power Control: Down/Off 2024-04-10T00:30:50.757 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:30:50.771 INFO:teuthology.orchestra.console.smithi081:Power off 2024-04-10T00:30:50.772 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T00:30:50.780 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off 2024-04-10T00:30:50.792 DEBUG:teuthology.orchestra.console.smithi081:power off output: Chassis Power Control: Down/Off 2024-04-10T00:30:50.792 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:30:50.810 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on 2024-04-10T00:30:50.818 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on 2024-04-10T00:30:50.881 INFO:teuthology.orchestra.console.smithi037:Power off completed 2024-04-10T00:30:51.041 INFO:teuthology.lock.ops:unlocked: smithi037.front.sepia.ceph.com 2024-04-10T00:30:54.813 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:30:54.824 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T00:30:54.927 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off 2024-04-10T00:30:54.934 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is off 2024-04-10T00:30:55.028 INFO:teuthology.orchestra.console.smithi045:Power off completed 2024-04-10T00:30:55.035 INFO:teuthology.orchestra.console.smithi081:Power off completed 2024-04-10T00:30:55.190 INFO:teuthology.lock.ops:unlocked: smithi045.front.sepia.ceph.com 2024-04-10T00:30:55.198 INFO:teuthology.lock.ops:unlocked: smithi081.front.sepia.ceph.com 2024-04-10T00:30:55.587 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/dispatcher/supervisor.py", line 58, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi037