2023-07-15T23:13:55.062 INFO:root:teuthology version: 0.0.1.dev147+gff9991b 2023-07-15T23:13:55.064 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-15T23:13:55.064 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-15T23:14:10.185 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f835f90e820>, signals=[15]) 2023-07-15T23:14:10.187 INFO:teuthology.lock.ops:Start node 'smithi153.front.sepia.ceph.com' reimaging 2023-07-15T23:14:10.187 INFO:teuthology.lock.ops:Updating [smithi153.front.sepia.ceph.com]: reset os type and version on server 2023-07-15T23:14:10.187 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2023-07-15T23:14:14.926 DEBUG:teuthology.provision.fog.smithi153:FOG request: GET /host {"name": "smithi153"} 2023-07-15T23:14:14.998 DEBUG:teuthology.provision.fog.smithi153:FOG request: GET /image {"name": "smithi_ubuntu_20.04"} 2023-07-15T23:14:15.044 DEBUG:teuthology.provision.fog.smithi153:Requesting image smithi_ubuntu_20.04 (ID 2) 2023-07-15T23:14:15.044 DEBUG:teuthology.provision.fog.smithi153:FOG request: PUT /host/258 {"imageID": 2} 2023-07-15T23:14:15.102 INFO:teuthology.provision.fog.smithi153:Scheduling deploy of ubuntu 20.04 2023-07-15T23:14:15.102 DEBUG:teuthology.provision.fog.smithi153:FOG request: GET /task/active None 2023-07-15T23:14:15.141 DEBUG:teuthology.provision.fog.smithi153:FOG request: GET /tasktype {"name": "deploy"} 2023-07-15T23:14:15.184 DEBUG:teuthology.provision.fog.smithi153:FOG request: POST /host/258/task {"taskTypeID": 1} 2023-07-15T23:14:15.236 DEBUG:teuthology.provision.fog.smithi153:FOG request: GET /task/active None 2023-07-15T23:14:15.322 INFO:teuthology.orchestra.console:Power off smithi153 2023-07-15T23:14:15.323 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-15T23:14:15.346 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T23:14:19.368 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T23:14:24.478 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T23:14:29.592 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T23:14:34.702 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T23:14:34.912 INFO:teuthology.orchestra.console:Power off for smithi153 completed 2023-07-15T23:14:35.013 INFO:teuthology.orchestra.console:Power on smithi153 2023-07-15T23:14:35.013 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-07-15T23:14:35.035 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T23:14:39.061 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T23:14:43.170 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T23:14:48.281 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T23:14:53.392 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T23:14:58.504 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T23:15:03.615 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T23:15:08.726 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T23:15:13.836 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T23:15:18.949 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T23:15:24.059 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T23:15:29.174 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T23:15:34.284 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T23:15:39.394 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T23:15:44.508 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T23:15:45.618 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds 2023-07-15T23:15:45.619 DEBUG:teuthology.provision.fog.smithi153:Canceling deploy task with ID 389258 2023-07-15T23:15:45.620 DEBUG:teuthology.provision.fog.smithi153:FOG request: DELETE /task/cancel {"id": 389258} 2023-07-15T23:15:50.678 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/orchestra/console.py", line 243, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi153 2023-07-15T23:15:50.679 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-15T23:15:50.679 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-15T23:15:52.318 INFO:teuthology.nuke:Checking targets against current locks 2023-07-15T23:15:52.369 DEBUG:teuthology.nuke:shortname: smithi153 2023-07-15T23:15:52.370 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-15T23:15:52.391 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-14_23:49:32-rbd-reef-distro-default-smithi/7339761', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-15 23:13:54.318808', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpNgV0EuyFWr8Bn57kmZRTjlLBGmfyJO0bUJxp4f5vXEKKetTTvEV48I8fVz3YnuPLjrKjNk+73y4PE8cI7TLhhzfwcFtL/YFitnH1WTBe+EUHQQw6dMZ2l8cTEmVTdkKuiHHJDO3WQPojMEQTmlxfF5bbt5GP+5+ZRfnpC5O4/MOjnQbGP2E9P1GmVtASEqxiRce5zUm36YP8IinFN7w6DOZS9kHcNzQIX0ycqtP4f7vNvT6f1B7LR5DKKTX7xAwvIELhdx3RpcZEPn9jTRan6JBTH/0mlCFhq/cCZfUWDejqjlMrfTEsYN/l1oSyeZU8e5c7cV26s+mJXtukO1HG9xq7a72luqWFNLY/S/9s+r+IJsSEzZelcLx43j98Zbihs4Y0M7GJ9Nxn+xxOfbUv8yiiXGYGrMLqmbW4uUJxM9qi98yMUIRfJlM4BVhJLADDT5PsvTeKFZtUrMlcoiUtJwWQ8MgjOPZaWfF4GwJd5WOJTiYltbXEYqNrznYjreM='} 2023-07-15T23:15:52.449 INFO:teuthology.orchestra.console:Power off smithi153 2023-07-15T23:15:52.450 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-15T23:16:00.257 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T23:16:04.284 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T23:16:09.397 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T23:16:14.508 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T23:16:19.619 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T23:16:24.729 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T23:16:24.941 INFO:teuthology.orchestra.console:Power off for smithi153 completed 2023-07-15T23:16:25.097 INFO:teuthology.lock.ops:unlocked: smithi153.front.sepia.ceph.com 2023-07-15T23:16:25.225 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/dispatcher/supervisor.py", line 57, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/orchestra/console.py", line 243, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi153