2024-02-09T07:34:32.049 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-09T07:34:32.049 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-09T07:34:32.259 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f73a39a8c10>, signals=[15]) 2024-02-09T07:34:32.260 INFO:teuthology.lock.ops:Start node 'smithi040.front.sepia.ceph.com' reimaging 2024-02-09T07:34:32.260 INFO:teuthology.lock.ops:Updating [smithi040.front.sepia.ceph.com]: reset os type and version on server 2024-02-09T07:34:32.260 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2024-02-09T07:34:32.283 INFO:teuthology.lock.ops:Start node 'smithi077.front.sepia.ceph.com' reimaging 2024-02-09T07:34:32.284 INFO:teuthology.lock.ops:Updating [smithi077.front.sepia.ceph.com]: reset os type and version on server 2024-02-09T07:34:32.284 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2024-02-09T07:34:32.308 INFO:teuthology.lock.ops:Start node 'smithi151.front.sepia.ceph.com' reimaging 2024-02-09T07:34:32.308 INFO:teuthology.lock.ops:Updating [smithi151.front.sepia.ceph.com]: reset os type and version on server 2024-02-09T07:34:32.308 INFO:teuthology.lock.ops:Updating smithi151.front.sepia.ceph.com on lock server 2024-02-09T07:34:32.333 INFO:teuthology.lock.ops:Start node 'smithi167.front.sepia.ceph.com' reimaging 2024-02-09T07:34:32.333 INFO:teuthology.lock.ops:Updating [smithi167.front.sepia.ceph.com]: reset os type and version on server 2024-02-09T07:34:32.334 INFO:teuthology.lock.ops:Updating smithi167.front.sepia.ceph.com on lock server 2024-02-09T07:34:32.405 DEBUG:teuthology.provision.fog.smithi040:Requesting image smithi_ubuntu_22.04 (ID 6) 2024-02-09T07:34:32.423 DEBUG:teuthology.provision.fog.smithi077:Requesting image smithi_ubuntu_22.04 (ID 6) 2024-02-09T07:34:32.453 DEBUG:teuthology.provision.fog.smithi151:Requesting image smithi_ubuntu_22.04 (ID 6) 2024-02-09T07:34:32.457 INFO:teuthology.provision.fog.smithi040:Scheduling deploy of ubuntu 22.04 2024-02-09T07:34:32.482 DEBUG:teuthology.provision.fog.smithi167:Requesting image smithi_ubuntu_22.04 (ID 6) 2024-02-09T07:34:32.485 INFO:teuthology.provision.fog.smithi077:Scheduling deploy of ubuntu 22.04 2024-02-09T07:34:32.504 INFO:teuthology.provision.fog.smithi151:Scheduling deploy of ubuntu 22.04 2024-02-09T07:34:32.525 INFO:teuthology.provision.fog.smithi167:Scheduling deploy of ubuntu 22.04 2024-02-09T07:34:32.714 INFO:teuthology.orchestra.console.smithi040:Power off 2024-02-09T07:34:32.714 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-09T07:34:32.738 DEBUG:teuthology.orchestra.console.smithi040:power off output: Chassis Power Control: Down/Off 2024-02-09T07:34:32.738 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:34:32.777 INFO:teuthology.orchestra.console.smithi151:Power off 2024-02-09T07:34:32.777 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-09T07:34:32.792 INFO:teuthology.orchestra.console.smithi077:Power off 2024-02-09T07:34:32.792 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-09T07:34:32.803 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2024-02-09T07:34:32.814 DEBUG:teuthology.orchestra.console.smithi077:power off output: Chassis Power Control: Down/Off 2024-02-09T07:34:32.814 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:34:32.833 INFO:teuthology.orchestra.console.smithi167:Power off 2024-02-09T07:34:32.834 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-09T07:34:32.843 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off 2024-02-09T07:34:32.843 DEBUG:teuthology.orchestra.console.smithi151:power off output: Chassis Power Control: Down/Off 2024-02-09T07:34:32.844 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:34:32.854 DEBUG:teuthology.orchestra.console.smithi167:power off output: Chassis Power Control: Down/Off 2024-02-09T07:34:32.855 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:34:32.869 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2024-02-09T07:34:32.882 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is off 2024-02-09T07:34:32.944 INFO:teuthology.orchestra.console.smithi077:Power off completed 2024-02-09T07:34:32.983 INFO:teuthology.orchestra.console.smithi167:Power off completed 2024-02-09T07:34:33.045 INFO:teuthology.orchestra.console.smithi077:Power on 2024-02-09T07:34:33.045 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-02-09T07:34:33.071 DEBUG:teuthology.orchestra.console.smithi077:power on output: Chassis Power Control: Up/On 2024-02-09T07:34:33.072 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:34:33.085 INFO:teuthology.orchestra.console.smithi167:Power on 2024-02-09T07:34:33.085 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-02-09T07:34:33.096 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off 2024-02-09T07:34:33.113 DEBUG:teuthology.orchestra.console.smithi167:power on output: Chassis Power Control: Up/On 2024-02-09T07:34:33.113 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:34:33.135 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is off 2024-02-09T07:34:36.808 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:34:36.870 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:34:36.917 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-02-09T07:34:36.979 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2024-02-09T07:34:37.019 INFO:teuthology.orchestra.console.smithi040:Power off completed 2024-02-09T07:34:37.098 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:34:37.120 INFO:teuthology.orchestra.console.smithi040:Power on 2024-02-09T07:34:37.120 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-02-09T07:34:37.136 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:34:37.147 DEBUG:teuthology.orchestra.console.smithi040:power on output: Chassis Power Control: Up/On 2024-02-09T07:34:37.148 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:34:37.170 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-02-09T07:34:37.207 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off 2024-02-09T07:34:37.248 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is off 2024-02-09T07:34:40.984 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:34:41.093 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2024-02-09T07:34:41.171 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:34:41.209 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:34:41.249 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:34:41.280 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-02-09T07:34:41.318 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-02-09T07:34:41.419 INFO:teuthology.orchestra.console.smithi077:Power on completed 2024-02-09T07:34:41.429 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is on 2024-02-09T07:34:41.521 INFO:teuthology.provision.fog.smithi077:Waiting for deploy to finish 2024-02-09T07:34:41.530 INFO:teuthology.orchestra.console.smithi167:Power on completed 2024-02-09T07:34:41.631 INFO:teuthology.provision.fog.smithi167:Waiting for deploy to finish 2024-02-09T07:34:45.096 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:34:45.207 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is off 2024-02-09T07:34:45.282 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:34:45.308 INFO:teuthology.orchestra.console.smithi151:Power off completed 2024-02-09T07:34:45.392 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-02-09T07:34:45.409 INFO:teuthology.orchestra.console.smithi151:Power on 2024-02-09T07:34:45.409 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-02-09T07:34:45.431 DEBUG:teuthology.orchestra.console.smithi151:power on output: Chassis Power Control: Up/On 2024-02-09T07:34:45.432 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:34:45.453 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is off 2024-02-09T07:34:49.396 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:34:49.454 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:34:49.506 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-02-09T07:34:49.564 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2024-02-09T07:34:49.665 INFO:teuthology.orchestra.console.smithi151:Power on completed 2024-02-09T07:34:49.766 INFO:teuthology.provision.fog.smithi151:Waiting for deploy to finish 2024-02-09T07:34:53.510 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:34:53.619 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-02-09T07:34:57.620 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:34:57.729 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-02-09T07:35:01.732 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:35:01.841 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-02-09T07:35:05.843 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:35:05.955 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-02-09T07:35:09.960 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:35:10.071 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-02-09T07:35:14.074 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:35:14.185 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-02-09T07:35:18.188 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:35:18.298 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-02-09T07:35:22.301 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:35:22.411 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-02-09T07:35:26.415 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:35:26.525 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-02-09T07:35:30.528 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:35:30.637 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-02-09T07:35:34.640 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:35:34.749 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-02-09T07:35:34.750 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds 2024-02-09T07:35:34.850 DEBUG:teuthology.provision.fog.smithi040:Canceling deploy task with ID 779827 2024-02-09T07:35:34.906 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi040 2024-02-09T07:35:34.907 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-09T07:35:35.213 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e166c5e5dea64824a8f959cf33d8f6e5 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi040 2024-02-09T07:35:35.215 INFO:teuthology.nuke:Checking targets against current locks 2024-02-09T07:35:35.281 DEBUG:teuthology.nuke:shortname: smithi040 2024-02-09T07:35:35.281 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-09T07:35:35.323 DEBUG:teuthology.nuke:shortname: smithi077 2024-02-09T07:35:35.323 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-09T07:35:35.325 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_00:15:46-rados-wip-yuri2-testing-2024-02-08-0727-distro-default-smithi/7553279', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-09 07:34:30.877413', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPUgrqpkQ3fjDvgg/5QvV3feyvxsiod1UJOzPiP/bSl6zfwftUOFxV0q/7C0AE01i1x9pEHdhO9cS8nBIVarJto='} 2024-02-09T07:35:35.354 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_00:15:46-rados-wip-yuri2-testing-2024-02-08-0727-distro-default-smithi/7553279', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-09 07:34:30.881658', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHSAlWjESMc0Gra5f/6dvvbjuYJc2qVK13Ztw1gt4S0hdg9pLl/eAcLsaHQz3OAGEzcbKS0DF4bknjuwe5HHhPE='} 2024-02-09T07:35:35.357 DEBUG:teuthology.nuke:shortname: smithi151 2024-02-09T07:35:35.358 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-09T07:35:35.390 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_00:15:46-rados-wip-yuri2-testing-2024-02-08-0727-distro-default-smithi/7553279', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-09 07:34:30.878938', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHOkN6aViO7D6wdU/q8iWrdnXT52LLmQnV/DCrX20VndjYe2DkVseYHgAlbZuekV6J8LpNrqMpR4HtmgRiX2mBc='} 2024-02-09T07:35:35.391 DEBUG:teuthology.nuke:shortname: smithi167 2024-02-09T07:35:35.391 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-09T07:35:35.398 INFO:teuthology.orchestra.console.smithi040:Power off 2024-02-09T07:35:35.398 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-09T07:35:35.424 DEBUG:teuthology.orchestra.console.smithi040:power off output: Chassis Power Control: Down/Off 2024-02-09T07:35:35.424 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:35:35.444 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi167.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_00:15:46-rados-wip-yuri2-testing-2024-02-08-0727-distro-default-smithi/7553279', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-09 07:34:30.880309', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKHaQ8fVVYkGMPjg1dboiFpLsX0IOF5ADMkKP/aj7+cnVY2MIOzi/bR/J04k2rE7Y86yxPe5JDg7ophOD5yUnWs='} 2024-02-09T07:35:35.451 INFO:teuthology.orchestra.console.smithi077:Power off 2024-02-09T07:35:35.452 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-09T07:35:35.465 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-02-09T07:35:35.490 INFO:teuthology.orchestra.console.smithi151:Power off 2024-02-09T07:35:35.491 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-09T07:35:35.514 DEBUG:teuthology.orchestra.console.smithi151:power off output: Chassis Power Control: Down/Off 2024-02-09T07:35:35.515 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:35:35.540 INFO:teuthology.orchestra.console.smithi167:Power off 2024-02-09T07:35:35.541 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-09T07:35:35.550 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2024-02-09T07:35:35.566 INFO:teuthology.orchestra.console.smithi040:Power off completed 2024-02-09T07:35:35.569 DEBUG:teuthology.orchestra.console.smithi167:power off output: Chassis Power Control: Down/Off 2024-02-09T07:35:35.569 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:35:35.605 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is on 2024-02-09T07:35:35.626 DEBUG:teuthology.orchestra.console.smithi077:power off output: Chassis Power Control: Down/Off 2024-02-09T07:35:35.626 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:35:35.652 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-02-09T07:35:35.723 INFO:teuthology.lock.ops:unlocked: smithi040.front.sepia.ceph.com 2024-02-09T07:35:39.552 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:35:39.606 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:35:39.653 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T07:35:39.663 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is off 2024-02-09T07:35:39.716 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is off 2024-02-09T07:35:39.764 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off 2024-02-09T07:35:39.765 INFO:teuthology.orchestra.console.smithi151:Power off completed 2024-02-09T07:35:39.817 INFO:teuthology.orchestra.console.smithi167:Power off completed 2024-02-09T07:35:39.864 INFO:teuthology.orchestra.console.smithi077:Power off completed 2024-02-09T07:35:39.922 INFO:teuthology.lock.ops:unlocked: smithi151.front.sepia.ceph.com 2024-02-09T07:35:39.962 INFO:teuthology.lock.ops:unlocked: smithi167.front.sepia.ceph.com 2024-02-09T07:35:40.012 INFO:teuthology.lock.ops:unlocked: smithi077.front.sepia.ceph.com 2024-02-09T07:35:40.532 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/dispatcher/supervisor.py", line 58, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi040