2023-06-28T10:16:01.232 INFO:root:teuthology version: 0.0.1.dev132+gaa9345a 2023-06-28T10:16:01.234 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-28T10:16:01.234 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-28T10:16:01.342 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/yuriw-2023-06-27_22:58:43-fs-reef-release-distro-default-smithi/jobs/ failed with status 500: Internal Server Error

Internal Server Error

2023-06-28T10:16:02.885 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbe2a2945e0>, signals=[15]) 2023-06-28T10:16:02.887 INFO:teuthology.lock.ops:Start node 'smithi047.front.sepia.ceph.com' reimaging 2023-06-28T10:16:02.888 INFO:teuthology.lock.ops:Updating [smithi047.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T10:16:02.888 INFO:teuthology.lock.ops:Updating smithi047.front.sepia.ceph.com on lock server 2023-06-28T10:16:02.910 INFO:teuthology.lock.ops:Start node 'smithi102.front.sepia.ceph.com' reimaging 2023-06-28T10:16:02.910 INFO:teuthology.lock.ops:Updating [smithi102.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T10:16:02.911 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server 2023-06-28T10:16:02.912 DEBUG:teuthology.provision.fog.smithi047:FOG request: GET /host {"name": "smithi047"} 2023-06-28T10:16:02.935 INFO:teuthology.lock.ops:Start node 'smithi160.front.sepia.ceph.com' reimaging 2023-06-28T10:16:02.936 INFO:teuthology.lock.ops:Updating [smithi160.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T10:16:02.936 INFO:teuthology.lock.ops:Updating smithi160.front.sepia.ceph.com on lock server 2023-06-28T10:16:02.937 DEBUG:teuthology.provision.fog.smithi102:FOG request: GET /host {"name": "smithi102"} 2023-06-28T10:16:02.960 DEBUG:teuthology.provision.fog.smithi160:FOG request: GET /host {"name": "smithi160"} 2023-06-28T10:16:02.992 DEBUG:teuthology.provision.fog.smithi047:FOG request: GET /image {"name": "smithi_rhel_8.6"} 2023-06-28T10:16:03.017 DEBUG:teuthology.provision.fog.smithi102:FOG request: GET /image {"name": "smithi_rhel_8.6"} 2023-06-28T10:16:03.031 DEBUG:teuthology.provision.fog.smithi047:Requesting image smithi_rhel_8.6 (ID 5) 2023-06-28T10:16:03.032 DEBUG:teuthology.provision.fog.smithi047:FOG request: PUT /host/152 {"imageID": 5} 2023-06-28T10:16:03.039 DEBUG:teuthology.provision.fog.smithi160:FOG request: GET /image {"name": "smithi_rhel_8.6"} 2023-06-28T10:16:03.057 DEBUG:teuthology.provision.fog.smithi102:Requesting image smithi_rhel_8.6 (ID 5) 2023-06-28T10:16:03.058 DEBUG:teuthology.provision.fog.smithi102:FOG request: PUT /host/207 {"imageID": 5} 2023-06-28T10:16:03.074 INFO:teuthology.provision.fog.smithi047:Scheduling deploy of rhel 8.6 2023-06-28T10:16:03.074 DEBUG:teuthology.provision.fog.smithi047:FOG request: GET /task/active None 2023-06-28T10:16:03.077 DEBUG:teuthology.provision.fog.smithi160:Requesting image smithi_rhel_8.6 (ID 5) 2023-06-28T10:16:03.077 DEBUG:teuthology.provision.fog.smithi160:FOG request: PUT /host/265 {"imageID": 5} 2023-06-28T10:16:03.103 INFO:teuthology.provision.fog.smithi102:Scheduling deploy of rhel 8.6 2023-06-28T10:16:03.103 DEBUG:teuthology.provision.fog.smithi102:FOG request: GET /task/active None 2023-06-28T10:16:03.119 INFO:teuthology.provision.fog.smithi160:Scheduling deploy of rhel 8.6 2023-06-28T10:16:03.120 DEBUG:teuthology.provision.fog.smithi160:FOG request: GET /task/active None 2023-06-28T10:16:03.139 DEBUG:teuthology.provision.fog.smithi047:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T10:16:03.167 DEBUG:teuthology.provision.fog.smithi102:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T10:16:03.175 DEBUG:teuthology.provision.fog.smithi047:FOG request: POST /host/152/task {"taskTypeID": 1} 2023-06-28T10:16:03.178 DEBUG:teuthology.provision.fog.smithi160:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T10:16:03.196 DEBUG:teuthology.provision.fog.smithi102:FOG request: POST /host/207/task {"taskTypeID": 1} 2023-06-28T10:16:03.220 DEBUG:teuthology.provision.fog.smithi160:FOG request: POST /host/265/task {"taskTypeID": 1} 2023-06-28T10:16:03.248 DEBUG:teuthology.provision.fog.smithi047:FOG request: GET /task/active None 2023-06-28T10:16:03.259 DEBUG:teuthology.provision.fog.smithi102:FOG request: GET /task/active None 2023-06-28T10:16:03.268 DEBUG:teuthology.provision.fog.smithi160:FOG request: GET /task/active None 2023-06-28T10:16:03.363 INFO:teuthology.orchestra.console:Power off smithi047 2023-06-28T10:16:03.363 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T10:16:03.378 INFO:teuthology.orchestra.console:Power off smithi160 2023-06-28T10:16:03.379 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T10:16:03.393 INFO:teuthology.orchestra.console:Power off smithi102 2023-06-28T10:16:03.394 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T10:16:03.403 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:03.412 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:03.424 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:07.436 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:07.444 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:07.452 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:12.548 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:12.556 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:12.564 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:17.660 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:17.671 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:17.679 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:22.774 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:22.782 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:22.792 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:27.885 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:27.894 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:27.902 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:28.097 INFO:teuthology.orchestra.console:Power off for smithi160 completed 2023-06-28T10:16:28.105 INFO:teuthology.orchestra.console:Power off for smithi047 completed 2023-06-28T10:16:28.198 INFO:teuthology.orchestra.console:Power on smithi160 2023-06-28T10:16:28.199 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T10:16:28.208 INFO:teuthology.orchestra.console:Power on smithi047 2023-06-28T10:16:28.208 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T10:16:33.012 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:33.222 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:33.235 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:37.260 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:38.125 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:38.231 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:42.371 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:43.243 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:43.342 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:47.484 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:48.355 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:48.453 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:52.598 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:53.468 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:53.565 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:57.708 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:58.581 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:58.677 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:16:58.890 INFO:teuthology.orchestra.console:Power on for smithi160 completed 2023-06-28T10:16:58.991 INFO:teuthology.provision.fog.smithi160:Waiting for deploy to finish 2023-06-28T10:16:58.991 DEBUG:teuthology.provision.fog.smithi160:FOG request: GET /task/active None 2023-06-28T10:17:02.821 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:17:03.694 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:17:07.932 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:17:08.808 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:17:13.044 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:17:13.919 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:17:14.063 DEBUG:teuthology.provision.fog.smithi160:FOG request: GET /task/active None 2023-06-28T10:17:24.172 INFO:teuthology.orchestra.console:Power off for smithi102 completed 2023-06-28T10:17:24.273 INFO:teuthology.orchestra.console:Power on smithi102 2023-06-28T10:17:24.273 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T10:17:28.075 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:17:33.187 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:17:38.297 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:17:39.143 DEBUG:teuthology.provision.fog.smithi160:FOG request: GET /task/active None 2023-06-28T10:17:44.147 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:17:48.149 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:17:49.160 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:17:53.264 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:17:54.271 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:17:58.379 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:17:59.236 DEBUG:teuthology.provision.fog.smithi160:FOG request: GET /task/active None 2023-06-28T10:18:04.241 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:18:04.250 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds 2023-06-28T10:18:04.251 DEBUG:teuthology.provision.fog.smithi047:Canceling deploy task with ID 350835 2023-06-28T10:18:04.252 DEBUG:teuthology.provision.fog.smithi047:FOG request: DELETE /task/cancel {"id": 350835} 2023-06-28T10:18:07.047 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/console.py", line 243, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi047 2023-06-28T10:18:07.048 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-28T10:18:07.048 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-28T10:18:12.052 INFO:teuthology.orchestra.console:Power on for smithi102 completed 2023-06-28T10:18:12.133 INFO:teuthology.nuke:Checking targets against current locks 2023-06-28T10:18:17.141 INFO:teuthology.provision.fog.smithi102:Waiting for deploy to finish 2023-06-28T10:18:17.141 DEBUG:teuthology.provision.fog.smithi102:FOG request: GET /task/active None 2023-06-28T10:18:32.185 DEBUG:teuthology.provision.fog.smithi160:FOG request: GET /task/active None 2023-06-28T10:18:47.205 DEBUG:teuthology.provision.fog.smithi102:FOG request: GET /task/active None 2023-06-28T10:18:47.208 DEBUG:teuthology.nuke:shortname: smithi047 2023-06-28T10:18:47.208 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T10:18:55.482 DEBUG:teuthology.nuke:shortname: smithi102 2023-06-28T10:18:55.483 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T10:18:55.510 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi047.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-27_22:58:43-fs-reef-release-distro-default-smithi/7319522', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 10:16:00.361136', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOJSMRdxVvy5VVCEHcmI1rs4k2mcc7TITDi/v/9PEMrkaxY/rcJ/yeHoeaVNfRFQDMMa13vafBLmSd2bqb4xBwk='} 2023-06-28T10:19:00.518 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-27_22:58:43-fs-reef-release-distro-default-smithi/7319522', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 10:16:00.359799', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGy9fh72ced61/1ffDxrP2KDAIksUlwNb//Mus0JzVbfSl0ZqxMukdPKM2RDWY1FjogMJI+zvk/G9UZx/MuLqf6VIOVKNQANA44+xkIACM49tg9HBElbnWG7bWwNEXxDs3jCLJpxcjEhtev9oD79o+Sqknr10cVgFOkXFoJ3Xsex+8TcIRHZb2Lbkypp8mdpFdkjCUL0bMmTzcjn6yC6mqvEvAjeloU+buUgrcCJ3w7wYYWEeIVlS7tGm3o4fUk150jwtC3PDZMF45F+6x2wPhSMFfGapFzmyN9vLbhvYUzvKUUGsdGYN+Qa6NbAmDYy42z3WaHG6o5AYONVQWPgC1Blf+wW2V62L/XemVvREdlGCX2sk15k0rHkkGs6N4iV69IRNkdI9b9J/2klJN27W/IUCYGhJgX8GM2wnt6wOEmnS1mXp3oe1rNoycIOwNacu2nTT4mzeUvUsFD/SIQ/PE09OaLBSfODR1LzBEtZ2gE8K/5DznL/bv3V1s9hx//Ks='} 2023-06-28T10:19:00.520 DEBUG:teuthology.nuke:shortname: smithi160 2023-06-28T10:19:00.520 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T10:19:10.528 DEBUG:teuthology.provision.fog.smithi160:FOG request: GET /task/active None 2023-06-28T10:19:10.529 DEBUG:teuthology.provision.fog.smithi102:FOG request: GET /task/active None 2023-06-28T10:19:21.250 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi160.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-27_22:58:43-fs-reef-release-distro-default-smithi/7319522', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 10:16:00.358315', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCyq5hkdrgQOrGLyiqnIoPBYjJsHySk+UAEwGX3Zf9m0dOqtbby32Rjx4BtgTywW9t3D1unKEfqnVVp/55fgdFFFErly7YIWob3tc0KZFkOsU+4gZtQCv8YL7j9POy7dzP4ZdDF+WTEfYIK7I2yMi1H0Z51R8QZSfirhY4F1oju29NchldxDNccOd6dVdgPjAj8OcSlIX+J23i9r2PgzvqJaGNXq7exyyGU0Mvr3oCbItXlTq3Hcj5Rd7FfUjc3JavPuyLiVa1Sp2R0s72RIKc3c/+QtGsxiHBUl8zGaNQTLtW77ElS/OCutZJtBxrGGwFb/Ym4hEyEgms/y5erKbkuq7+VP2Ip5zSVRUJzRHpi83xITRUgDs1rx4CJvDuwRuMJlA4MxkZCeUvwjx47y8I4hzR5tLpIbgX89z3Be6uTK5U0rmVWIa+WLNMkei1IKvi29FQt3NpM2dCfTRdk5VqMW+2YoBaBOCCEAM9oMa1b3vKS7qCMknthpGtLA7PVg+M='} 2023-06-28T10:19:21.282 INFO:teuthology.orchestra.console:Power off smithi102 2023-06-28T10:19:21.282 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T10:19:21.300 INFO:teuthology.orchestra.console:Power off smithi047 2023-06-28T10:19:21.301 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T10:19:21.312 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:19:21.324 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:19:21.342 INFO:teuthology.orchestra.console:Power off smithi160 2023-06-28T10:19:21.343 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T10:19:25.356 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:19:26.336 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:19:26.369 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:19:29.470 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:19:31.380 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:19:31.447 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:19:34.580 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:19:36.250 DEBUG:teuthology.provision.fog.smithi160:FOG request: GET /task/active None 2023-06-28T10:19:41.257 DEBUG:teuthology.provision.fog.smithi102:FOG request: GET /task/active None 2023-06-28T10:19:41.259 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:19:41.266 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:19:41.274 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:19:46.365 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T10:19:46.387 INFO:teuthology.orchestra.console:Power off for smithi160 completed 2023-06-28T10:19:46.399 INFO:teuthology.orchestra.console:Power off for smithi102 completed 2023-06-28T10:19:49.655 WARNING:teuthology.provision.fog.smithi160:[Errno None] Unable to connect to port 22 on 172.21.15.160 2023-06-28T10:19:54.657 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T10:19:59.698 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T10:20:09.704 DEBUG:teuthology.provision.fog.smithi102:FOG request: GET /task/active None 2023-06-28T10:20:09.706 WARNING:teuthology.provision.fog.smithi160:[Errno None] Unable to connect to port 22 on 172.21.15.160 2023-06-28T10:20:14.708 INFO:teuthology.orchestra.console:Power off for smithi047 completed 2023-06-28T10:20:14.710 INFO:teuthology.lock.ops:unlocked: smithi102.front.sepia.ceph.com 2023-06-28T10:20:14.738 INFO:teuthology.lock.ops:unlocked: smithi160.front.sepia.ceph.com 2023-06-28T10:20:19.817 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T10:20:24.845 WARNING:teuthology.provision.fog.smithi160:[Errno None] Unable to connect to port 22 on 172.21.15.160 2023-06-28T10:20:24.870 INFO:teuthology.lock.ops:unlocked: smithi047.front.sepia.ceph.com 2023-06-28T10:20:29.876 DEBUG:teuthology.provision.fog.smithi102:FOG request: GET /task/active None 2023-06-28T10:20:34.881 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T10:20:45.062 WARNING:teuthology.provision.fog.smithi160:[Errno None] Unable to connect to port 22 on 172.21.15.160 2023-06-28T10:20:50.315 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 57, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/console.py", line 243, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi047