2024-01-03T14:49:14.465 INFO:root:teuthology version: 0.0.1.dev228+g01dd171 2024-01-03T14:49:14.466 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-03T14:49:14.613 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6f43f6e940>, signals=[15]) 2024-01-03T14:49:14.614 INFO:teuthology.lock.ops:Start node 'smithi007.front.sepia.ceph.com' reimaging 2024-01-03T14:49:14.614 INFO:teuthology.lock.ops:Updating [smithi007.front.sepia.ceph.com]: reset os type and version on server 2024-01-03T14:49:14.614 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2024-01-03T14:49:14.634 INFO:teuthology.lock.ops:Start node 'smithi043.front.sepia.ceph.com' reimaging 2024-01-03T14:49:14.634 INFO:teuthology.lock.ops:Updating [smithi043.front.sepia.ceph.com]: reset os type and version on server 2024-01-03T14:49:14.635 INFO:teuthology.lock.ops:Updating smithi043.front.sepia.ceph.com on lock server 2024-01-03T14:49:14.659 INFO:teuthology.lock.ops:Start node 'smithi069.front.sepia.ceph.com' reimaging 2024-01-03T14:49:14.659 INFO:teuthology.lock.ops:Updating [smithi069.front.sepia.ceph.com]: reset os type and version on server 2024-01-03T14:49:14.659 INFO:teuthology.lock.ops:Updating smithi069.front.sepia.ceph.com on lock server 2024-01-03T14:49:14.770 DEBUG:teuthology.provision.fog.smithi007:Requesting image smithi_rhel_8.6 (ID 5) 2024-01-03T14:49:14.780 DEBUG:teuthology.provision.fog.smithi043:Requesting image smithi_rhel_8.6 (ID 5) 2024-01-03T14:49:14.791 DEBUG:teuthology.provision.fog.smithi069:Requesting image smithi_rhel_8.6 (ID 5) 2024-01-03T14:49:14.806 INFO:teuthology.provision.fog.smithi007:Scheduling deploy of rhel 8.6 2024-01-03T14:49:14.827 INFO:teuthology.provision.fog.smithi069:Scheduling deploy of rhel 8.6 2024-01-03T14:49:14.830 INFO:teuthology.provision.fog.smithi043:Scheduling deploy of rhel 8.6 2024-01-03T14:49:15.057 INFO:teuthology.orchestra.console.smithi007:Power off 2024-01-03T14:49:15.058 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-03T14:49:15.079 DEBUG:teuthology.orchestra.console.smithi007:power off output: Chassis Power Control: Down/Off 2024-01-03T14:49:15.079 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:49:15.092 INFO:teuthology.orchestra.console.smithi069:Power off 2024-01-03T14:49:15.092 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-03T14:49:15.107 INFO:teuthology.orchestra.console.smithi043:Power off 2024-01-03T14:49:15.107 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-03T14:49:15.115 DEBUG:teuthology.orchestra.console.smithi069:power off output: Chassis Power Control: Down/Off 2024-01-03T14:49:15.115 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:49:15.122 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2024-01-03T14:49:15.128 DEBUG:teuthology.orchestra.console.smithi043:power off output: Chassis Power Control: Down/Off 2024-01-03T14:49:15.128 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:49:15.136 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is off 2024-01-03T14:49:15.149 DEBUG:teuthology.orchestra.console.smithi043:check power output: Chassis Power is off 2024-01-03T14:49:15.237 INFO:teuthology.orchestra.console.smithi069:Power off completed 2024-01-03T14:49:15.250 INFO:teuthology.orchestra.console.smithi043:Power off completed 2024-01-03T14:49:15.338 INFO:teuthology.orchestra.console.smithi069:Power on 2024-01-03T14:49:15.338 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-01-03T14:49:15.350 INFO:teuthology.orchestra.console.smithi043:Power on 2024-01-03T14:49:15.351 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-01-03T14:49:15.362 DEBUG:teuthology.orchestra.console.smithi069:power on output: Chassis Power Control: Up/On 2024-01-03T14:49:15.362 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:49:15.370 DEBUG:teuthology.orchestra.console.smithi043:power on output: Chassis Power Control: Up/On 2024-01-03T14:49:15.370 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:49:15.384 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is off 2024-01-03T14:49:15.391 DEBUG:teuthology.orchestra.console.smithi043:check power output: Chassis Power is off 2024-01-03T14:49:19.124 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:49:19.233 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2024-01-03T14:49:19.385 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:49:19.392 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:49:19.494 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is off 2024-01-03T14:49:19.500 DEBUG:teuthology.orchestra.console.smithi043:check power output: Chassis Power is off 2024-01-03T14:49:23.238 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:49:23.346 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2024-01-03T14:49:23.495 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:49:23.502 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:49:23.604 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-01-03T14:49:23.705 INFO:teuthology.orchestra.console.smithi069:Power on completed 2024-01-03T14:49:23.728 DEBUG:teuthology.orchestra.console.smithi043:check power output: Chassis Power is on 2024-01-03T14:49:23.806 INFO:teuthology.provision.fog.smithi069:Waiting for deploy to finish 2024-01-03T14:49:23.830 INFO:teuthology.orchestra.console.smithi043:Power on completed 2024-01-03T14:49:23.931 INFO:teuthology.provision.fog.smithi043:Waiting for deploy to finish 2024-01-03T14:49:27.351 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:49:27.461 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is off 2024-01-03T14:49:27.562 INFO:teuthology.orchestra.console.smithi007:Power off completed 2024-01-03T14:49:27.662 INFO:teuthology.orchestra.console.smithi007:Power on 2024-01-03T14:49:27.663 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-01-03T14:49:27.687 DEBUG:teuthology.orchestra.console.smithi007:power on output: Chassis Power Control: Up/On 2024-01-03T14:49:27.688 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:49:27.711 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is off 2024-01-03T14:49:31.712 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:49:32.823 DEBUG:teuthology.orchestra.console.smithi007:check power output: 2024-01-03T14:49:36.824 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:49:37.934 DEBUG:teuthology.orchestra.console.smithi007:check power output: 2024-01-03T14:49:41.938 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:49:43.049 DEBUG:teuthology.orchestra.console.smithi007:check power output: 2024-01-03T14:49:47.054 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:49:48.163 DEBUG:teuthology.orchestra.console.smithi007:check power output: 2024-01-03T14:49:52.167 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:49:53.276 DEBUG:teuthology.orchestra.console.smithi007:check power output: 2024-01-03T14:49:57.280 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:49:58.392 DEBUG:teuthology.orchestra.console.smithi007:check power output: 2024-01-03T14:50:02.397 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:50:03.508 DEBUG:teuthology.orchestra.console.smithi007:check power output: 2024-01-03T14:50:07.513 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:50:08.624 DEBUG:teuthology.orchestra.console.smithi007:check power output: 2024-01-03T14:50:12.628 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:50:13.739 DEBUG:teuthology.orchestra.console.smithi007:check power output: 2024-01-03T14:50:17.744 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:50:18.852 DEBUG:teuthology.orchestra.console.smithi007:check power output: 2024-01-03T14:50:22.856 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:50:23.966 DEBUG:teuthology.orchestra.console.smithi007:check power output: 2024-01-03T14:50:27.971 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:50:29.043 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is off 2024-01-03T14:50:33.045 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:50:33.154 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is off 2024-01-03T14:50:37.156 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:50:37.265 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is off 2024-01-03T14:50:37.266 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds 2024-01-03T14:50:37.366 DEBUG:teuthology.provision.fog.smithi007:Canceling deploy task with ID 694638 2024-01-03T14:50:37.472 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi007 2024-01-03T14:50:37.474 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-03T14:50:37.618 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=30fe30f941af4580a033efe5972dd71c Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi007 2024-01-03T14:50:37.619 INFO:teuthology.nuke:Checking targets against current locks 2024-01-03T14:50:37.694 DEBUG:teuthology.nuke:shortname: smithi007 2024-01-03T14:50:37.694 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-03T14:50:37.717 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2024-01-03_14:48:42-fs-wip-choffman-62793-distro-default-smithi/7505156', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-03 14:49:13.625530', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOb7m9mihPlHJniVLWsYJ0UjvLbPeOsM0excUW/UnY17wIfOaGi/lkFloF4OAXJ7UHvZdICgfNngM317xpjf2uQ='} 2024-01-03T14:50:37.718 DEBUG:teuthology.nuke:shortname: smithi043 2024-01-03T14:50:37.718 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-03T14:50:37.743 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi043.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2024-01-03_14:48:42-fs-wip-choffman-62793-distro-default-smithi/7505156', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-03 14:49:13.624086', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMKMLFqO5k8z3Ah+he0xFjgb9iVczDi6jI0nHddTqYKBSGs66skReYaQTEB3SG+2Th6/lS+YjNke3fe4qGqO+PY='} 2024-01-03T14:50:37.747 DEBUG:teuthology.nuke:shortname: smithi069 2024-01-03T14:50:37.747 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-03T14:50:37.771 INFO:teuthology.orchestra.console.smithi007:Power off 2024-01-03T14:50:37.771 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-03T14:50:37.779 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2024-01-03_14:48:42-fs-wip-choffman-62793-distro-default-smithi/7505156', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-03 14:49:13.622423', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEPV3cml5ApzP5wwc6RmEAz1QK2nQ8Eqn5O8VF8fSsO3o1uOxGQyILkYE1bTW+UDa7fk+zQEivKodpevAPeyU9E='} 2024-01-03T14:50:37.792 DEBUG:teuthology.orchestra.console.smithi007:power off output: Chassis Power Control: Down/Off 2024-01-03T14:50:37.793 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:50:37.817 INFO:teuthology.orchestra.console.smithi043:Power off 2024-01-03T14:50:37.817 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-03T14:50:37.825 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is off 2024-01-03T14:50:37.841 DEBUG:teuthology.orchestra.console.smithi043:power off output: Chassis Power Control: Down/Off 2024-01-03T14:50:37.841 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:50:37.853 INFO:teuthology.orchestra.console.smithi069:Power off 2024-01-03T14:50:37.854 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-03T14:50:37.862 DEBUG:teuthology.orchestra.console.smithi043:check power output: Chassis Power is on 2024-01-03T14:50:37.875 DEBUG:teuthology.orchestra.console.smithi069:power off output: Chassis Power Control: Down/Off 2024-01-03T14:50:37.875 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:50:37.894 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-01-03T14:50:37.926 INFO:teuthology.orchestra.console.smithi007:Power off completed 2024-01-03T14:50:38.075 INFO:teuthology.lock.ops:unlocked: smithi007.front.sepia.ceph.com 2024-01-03T14:50:41.865 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:50:41.896 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:50:41.977 DEBUG:teuthology.orchestra.console.smithi043:check power output: Chassis Power is on 2024-01-03T14:50:42.007 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-01-03T14:50:45.982 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:50:46.008 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:50:46.094 DEBUG:teuthology.orchestra.console.smithi043:check power output: Chassis Power is on 2024-01-03T14:50:46.118 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is off 2024-01-03T14:50:46.219 INFO:teuthology.orchestra.console.smithi069:Power off completed 2024-01-03T14:50:46.374 INFO:teuthology.lock.ops:unlocked: smithi069.front.sepia.ceph.com 2024-01-03T14:50:50.096 DEBUG:teuthology.orchestra.console.smithi043:pexpect command: ipmitool -H smithi043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-03T14:50:50.207 DEBUG:teuthology.orchestra.console.smithi043:check power output: Chassis Power is off 2024-01-03T14:50:50.308 INFO:teuthology.orchestra.console.smithi043:Power off completed 2024-01-03T14:50:50.467 INFO:teuthology.lock.ops:unlocked: smithi043.front.sepia.ceph.com 2024-01-03T14:50:50.912 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/dispatcher/supervisor.py", line 58, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi007