2024-03-30T01:52:16.399 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-30T01:52:16.400 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-30T01:52:16.636 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff55729fc10>, signals=[15]) 2024-03-30T01:52:16.637 INFO:teuthology.lock.ops:Start node 'smithi057.front.sepia.ceph.com' reimaging 2024-03-30T01:52:16.637 INFO:teuthology.lock.ops:Updating [smithi057.front.sepia.ceph.com]: reset os type and version on server 2024-03-30T01:52:16.637 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server 2024-03-30T01:52:16.660 INFO:teuthology.lock.ops:Start node 'smithi115.front.sepia.ceph.com' reimaging 2024-03-30T01:52:16.660 INFO:teuthology.lock.ops:Updating [smithi115.front.sepia.ceph.com]: reset os type and version on server 2024-03-30T01:52:16.660 INFO:teuthology.lock.ops:Updating smithi115.front.sepia.ceph.com on lock server 2024-03-30T01:52:16.684 INFO:teuthology.lock.ops:Start node 'smithi150.front.sepia.ceph.com' reimaging 2024-03-30T01:52:16.684 INFO:teuthology.lock.ops:Updating [smithi150.front.sepia.ceph.com]: reset os type and version on server 2024-03-30T01:52:16.684 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2024-03-30T01:52:16.709 INFO:teuthology.lock.ops:Start node 'smithi203.front.sepia.ceph.com' reimaging 2024-03-30T01:52:16.709 INFO:teuthology.lock.ops:Updating [smithi203.front.sepia.ceph.com]: reset os type and version on server 2024-03-30T01:52:16.710 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server 2024-03-30T01:52:16.781 DEBUG:teuthology.provision.fog.smithi057:Requesting image smithi_ubuntu_22.04 (ID 6) 2024-03-30T01:52:16.810 DEBUG:teuthology.provision.fog.smithi115:Requesting image smithi_ubuntu_22.04 (ID 6) 2024-03-30T01:52:16.835 INFO:teuthology.provision.fog.smithi057:Scheduling deploy of ubuntu 22.04 2024-03-30T01:52:16.838 DEBUG:teuthology.provision.fog.smithi150:Requesting image smithi_ubuntu_22.04 (ID 6) 2024-03-30T01:52:16.847 DEBUG:teuthology.provision.fog.smithi203:Requesting image smithi_ubuntu_22.04 (ID 6) 2024-03-30T01:52:16.868 INFO:teuthology.provision.fog.smithi115:Scheduling deploy of ubuntu 22.04 2024-03-30T01:52:16.878 INFO:teuthology.provision.fog.smithi150:Scheduling deploy of ubuntu 22.04 2024-03-30T01:52:16.904 INFO:teuthology.provision.fog.smithi203:Scheduling deploy of ubuntu 22.04 2024-03-30T01:52:17.154 INFO:teuthology.orchestra.console.smithi057:Power off 2024-03-30T01:52:17.154 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T01:52:17.171 INFO:teuthology.orchestra.console.smithi115:Power off 2024-03-30T01:52:17.171 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T01:52:17.188 DEBUG:teuthology.orchestra.console.smithi057:power off output: Chassis Power Control: Down/Off 2024-03-30T01:52:17.189 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:52:17.198 DEBUG:teuthology.orchestra.console.smithi115:power off output: Chassis Power Control: Down/Off 2024-03-30T01:52:17.198 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:52:17.210 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on 2024-03-30T01:52:17.220 INFO:teuthology.orchestra.console.smithi203:Power off 2024-03-30T01:52:17.220 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T01:52:17.235 INFO:teuthology.orchestra.console.smithi150:Power off 2024-03-30T01:52:17.236 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T01:52:17.244 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is off 2024-03-30T01:52:17.246 DEBUG:teuthology.orchestra.console.smithi203:power off output: Chassis Power Control: Down/Off 2024-03-30T01:52:17.246 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:52:17.258 DEBUG:teuthology.orchestra.console.smithi150:power off output: Chassis Power Control: Down/Off 2024-03-30T01:52:17.259 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:52:17.268 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-03-30T01:52:17.281 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is off 2024-03-30T01:52:17.346 INFO:teuthology.orchestra.console.smithi115:Power off completed 2024-03-30T01:52:17.382 INFO:teuthology.orchestra.console.smithi150:Power off completed 2024-03-30T01:52:17.447 INFO:teuthology.orchestra.console.smithi115:Power on 2024-03-30T01:52:17.447 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-30T01:52:17.469 DEBUG:teuthology.orchestra.console.smithi115:power on output: Chassis Power Control: Up/On 2024-03-30T01:52:17.469 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:52:17.483 INFO:teuthology.orchestra.console.smithi150:Power on 2024-03-30T01:52:17.483 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-30T01:52:17.490 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is off 2024-03-30T01:52:17.503 DEBUG:teuthology.orchestra.console.smithi150:power on output: Chassis Power Control: Up/On 2024-03-30T01:52:17.504 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:52:17.525 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is off 2024-03-30T01:52:21.214 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:52:21.270 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:52:21.324 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on 2024-03-30T01:52:21.383 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-03-30T01:52:21.484 INFO:teuthology.orchestra.console.smithi203:Power off completed 2024-03-30T01:52:21.491 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:52:21.526 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:52:21.585 INFO:teuthology.orchestra.console.smithi203:Power on 2024-03-30T01:52:21.586 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-30T01:52:21.603 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is off 2024-03-30T01:52:21.611 DEBUG:teuthology.orchestra.console.smithi203:power on output: Chassis Power Control: Up/On 2024-03-30T01:52:21.611 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:52:21.636 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is off 2024-03-30T01:52:21.644 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-03-30T01:52:25.326 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:52:25.435 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on 2024-03-30T01:52:25.605 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:52:25.638 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:52:25.645 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:52:25.715 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is on 2024-03-30T01:52:25.747 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2024-03-30T01:52:25.755 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-03-30T01:52:25.816 INFO:teuthology.orchestra.console.smithi115:Power on completed 2024-03-30T01:52:25.847 INFO:teuthology.orchestra.console.smithi150:Power on completed 2024-03-30T01:52:25.917 INFO:teuthology.provision.fog.smithi115:Waiting for deploy to finish 2024-03-30T01:52:25.949 INFO:teuthology.provision.fog.smithi150:Waiting for deploy to finish 2024-03-30T01:52:29.438 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:52:29.547 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is off 2024-03-30T01:52:29.648 INFO:teuthology.orchestra.console.smithi057:Power off completed 2024-03-30T01:52:29.749 INFO:teuthology.orchestra.console.smithi057:Power on 2024-03-30T01:52:29.749 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-30T01:52:29.757 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:52:29.771 DEBUG:teuthology.orchestra.console.smithi057:power on output: Chassis Power Control: Up/On 2024-03-30T01:52:29.771 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:52:29.793 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is off 2024-03-30T01:52:29.865 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-03-30T01:52:33.798 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:52:33.866 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:52:33.907 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on 2024-03-30T01:52:33.975 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-03-30T01:52:34.008 INFO:teuthology.orchestra.console.smithi057:Power on completed 2024-03-30T01:52:34.109 INFO:teuthology.provision.fog.smithi057:Waiting for deploy to finish 2024-03-30T01:52:37.979 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:52:38.088 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-03-30T01:52:42.090 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:52:42.203 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-03-30T01:52:46.207 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:52:46.316 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-03-30T01:52:50.319 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:52:50.427 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-03-30T01:52:54.430 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:52:54.539 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-03-30T01:52:58.543 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:52:58.652 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-03-30T01:53:02.654 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:53:02.763 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-03-30T01:53:06.766 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:53:06.875 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-03-30T01:53:10.878 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:53:10.989 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-03-30T01:53:14.993 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:53:15.103 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-03-30T01:53:19.109 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:53:19.217 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-03-30T01:53:19.218 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds 2024-03-30T01:53:19.319 DEBUG:teuthology.provision.fog.smithi203:Canceling deploy task with ID 906534 2024-03-30T01:53:19.429 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi203 2024-03-30T01:53:19.431 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-30T01:53:21.705 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c7685bf9a0644afba98809dd26595c15 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi203 2024-03-30T01:53:21.707 INFO:teuthology.nuke:Checking targets against current locks 2024-03-30T01:53:21.771 DEBUG:teuthology.nuke:shortname: smithi057 2024-03-30T01:53:21.771 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T01:53:21.791 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi057.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629354', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 01:52:15.265313', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK4HuB0BGeSsqNT9ukTz2qgV59wLDc0Ug+wLZBPg/3UB5CWUgRLJiCGYEwumNZ8/W5WTjCulnGAEKnzJVs2srUU='} 2024-03-30T01:53:21.795 DEBUG:teuthology.nuke:shortname: smithi115 2024-03-30T01:53:21.795 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T01:53:21.822 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi115.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629354', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 01:52:15.263701', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2024-03-30T01:53:21.826 DEBUG:teuthology.nuke:shortname: smithi150 2024-03-30T01:53:21.826 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T01:53:21.853 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629354', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 01:52:15.262117', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2024-03-30T01:53:21.857 DEBUG:teuthology.nuke:shortname: smithi203 2024-03-30T01:53:21.857 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T01:53:21.863 INFO:teuthology.orchestra.console.smithi057:Power off 2024-03-30T01:53:21.863 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T01:53:21.880 INFO:teuthology.orchestra.console.smithi115:Power off 2024-03-30T01:53:21.880 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T01:53:21.889 DEBUG:teuthology.orchestra.console.smithi057:power off output: Chassis Power Control: Down/Off 2024-03-30T01:53:21.890 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:53:21.903 DEBUG:teuthology.orchestra.console.smithi115:power off output: Chassis Power Control: Down/Off 2024-03-30T01:53:21.903 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:53:21.914 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629354', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 01:52:15.260309', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL1Wi1de10qKuDe/XsCmBbgC/uWcKW5+1yUGDb5keKB8jP6/p83tn7qS9WGrU31FuvMzOo7toBjH5cgzxziTiQY='} 2024-03-30T01:53:21.918 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on 2024-03-30T01:53:21.928 INFO:teuthology.orchestra.console.smithi150:Power off 2024-03-30T01:53:21.929 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T01:53:21.942 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is on 2024-03-30T01:53:21.953 DEBUG:teuthology.orchestra.console.smithi150:power off output: Chassis Power Control: Down/Off 2024-03-30T01:53:21.953 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:53:21.968 INFO:teuthology.orchestra.console.smithi203:Power off 2024-03-30T01:53:21.969 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T01:53:21.979 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2024-03-30T01:53:21.991 DEBUG:teuthology.orchestra.console.smithi203:power off output: Chassis Power Control: Down/Off 2024-03-30T01:53:21.991 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:53:22.012 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-03-30T01:53:22.113 INFO:teuthology.orchestra.console.smithi203:Power off completed 2024-03-30T01:53:22.260 INFO:teuthology.lock.ops:unlocked: smithi203.front.sepia.ceph.com 2024-03-30T01:53:25.922 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:53:25.943 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:53:25.981 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:53:26.033 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is off 2024-03-30T01:53:26.052 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is off 2024-03-30T01:53:26.090 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2024-03-30T01:53:26.134 INFO:teuthology.orchestra.console.smithi057:Power off completed 2024-03-30T01:53:26.153 INFO:teuthology.orchestra.console.smithi115:Power off completed 2024-03-30T01:53:26.276 INFO:teuthology.lock.ops:unlocked: smithi057.front.sepia.ceph.com 2024-03-30T01:53:26.295 INFO:teuthology.lock.ops:unlocked: smithi115.front.sepia.ceph.com 2024-03-30T01:53:30.095 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T01:53:30.205 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is off 2024-03-30T01:53:30.306 INFO:teuthology.orchestra.console.smithi150:Power off completed 2024-03-30T01:53:30.461 INFO:teuthology.lock.ops:unlocked: smithi150.front.sepia.ceph.com 2024-03-30T01:53:30.973 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 58, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi203