2023-08-16T02:21:32.755 INFO:root:teuthology version: 0.0.1.dev183+g5435569 2023-08-16T02:21:32.756 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-16T02:21:42.353 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fee7e8d7e50>, signals=[15]) 2023-08-16T02:21:42.355 INFO:teuthology.lock.ops:Start node 'smithi077.front.sepia.ceph.com' reimaging 2023-08-16T02:21:42.355 INFO:teuthology.lock.ops:Updating [smithi077.front.sepia.ceph.com]: reset os type and version on server 2023-08-16T02:21:42.356 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2023-08-16T02:21:49.295 INFO:teuthology.lock.ops:Start node 'smithi081.front.sepia.ceph.com' reimaging 2023-08-16T02:21:49.296 INFO:teuthology.lock.ops:Updating [smithi081.front.sepia.ceph.com]: reset os type and version on server 2023-08-16T02:21:49.296 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2023-08-16T02:21:49.297 DEBUG:teuthology.provision.fog.smithi077:FOG request: GET /host {"name": "smithi077"} 2023-08-16T02:22:07.354 INFO:teuthology.lock.ops:Start node 'smithi148.front.sepia.ceph.com' reimaging 2023-08-16T02:22:07.354 INFO:teuthology.lock.ops:Updating [smithi148.front.sepia.ceph.com]: reset os type and version on server 2023-08-16T02:22:07.354 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2023-08-16T02:22:07.356 DEBUG:teuthology.provision.fog.smithi081:FOG request: GET /host {"name": "smithi081"} 2023-08-16T02:22:07.383 DEBUG:teuthology.provision.fog.smithi077:FOG request: GET /image {"name": "smithi_rhel_8.6"} 2023-08-16T02:22:07.420 DEBUG:teuthology.provision.fog.smithi148:FOG request: GET /host {"name": "smithi148"} 2023-08-16T02:22:07.424 DEBUG:teuthology.provision.fog.smithi077:Requesting image smithi_rhel_8.6 (ID 5) 2023-08-16T02:22:07.424 DEBUG:teuthology.provision.fog.smithi077:FOG request: PUT /host/182 {"imageID": 5} 2023-08-16T02:22:07.427 DEBUG:teuthology.provision.fog.smithi081:FOG request: GET /image {"name": "smithi_rhel_8.6"} 2023-08-16T02:22:07.475 INFO:teuthology.provision.fog.smithi077:Scheduling deploy of rhel 8.6 2023-08-16T02:22:07.476 DEBUG:teuthology.provision.fog.smithi077:FOG request: GET /task/active None 2023-08-16T02:22:07.479 DEBUG:teuthology.provision.fog.smithi148:FOG request: GET /image {"name": "smithi_rhel_8.6"} 2023-08-16T02:22:07.484 DEBUG:teuthology.provision.fog.smithi081:Requesting image smithi_rhel_8.6 (ID 5) 2023-08-16T02:22:07.484 DEBUG:teuthology.provision.fog.smithi081:FOG request: PUT /host/186 {"imageID": 5} 2023-08-16T02:22:07.522 DEBUG:teuthology.provision.fog.smithi148:Requesting image smithi_rhel_8.6 (ID 5) 2023-08-16T02:22:07.523 DEBUG:teuthology.provision.fog.smithi148:FOG request: PUT /host/253 {"imageID": 5} 2023-08-16T02:22:12.531 DEBUG:teuthology.provision.fog.smithi077:FOG request: GET /tasktype {"name": "deploy"} 2023-08-16T02:22:12.532 INFO:teuthology.provision.fog.smithi081:Scheduling deploy of rhel 8.6 2023-08-16T02:22:12.532 DEBUG:teuthology.provision.fog.smithi081:FOG request: GET /task/active None 2023-08-16T02:22:12.571 DEBUG:teuthology.provision.fog.smithi077:FOG request: POST /host/182/task {"taskTypeID": 1} 2023-08-16T02:22:12.588 INFO:teuthology.provision.fog.smithi148:Scheduling deploy of rhel 8.6 2023-08-16T02:22:12.589 DEBUG:teuthology.provision.fog.smithi148:FOG request: GET /task/active None 2023-08-16T02:22:12.606 DEBUG:teuthology.provision.fog.smithi081:FOG request: GET /tasktype {"name": "deploy"} 2023-08-16T02:22:12.632 DEBUG:teuthology.provision.fog.smithi077:FOG request: GET /task/active None 2023-08-16T02:22:12.648 DEBUG:teuthology.provision.fog.smithi081:FOG request: POST /host/186/task {"taskTypeID": 1} 2023-08-16T02:22:12.671 DEBUG:teuthology.provision.fog.smithi148:FOG request: GET /tasktype {"name": "deploy"} 2023-08-16T02:22:12.706 DEBUG:teuthology.provision.fog.smithi081:FOG request: GET /task/active None 2023-08-16T02:22:17.693 DEBUG:teuthology.provision.fog.smithi148:FOG request: POST /host/253/task {"taskTypeID": 1} 2023-08-16T02:22:17.725 INFO:teuthology.orchestra.console.smithi077:Power off 2023-08-16T02:22:17.726 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-16T02:22:17.749 DEBUG:teuthology.orchestra.console.smithi077:power off output: Chassis Power Control: Down/Off 2023-08-16T02:22:17.750 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:22:17.767 DEBUG:teuthology.provision.fog.smithi148:FOG request: GET /task/active None 2023-08-16T02:22:17.772 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2023-08-16T02:22:17.790 INFO:teuthology.orchestra.console.smithi081:Power off 2023-08-16T02:22:17.791 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-16T02:22:17.811 DEBUG:teuthology.orchestra.console.smithi081:power off output: Chassis Power Control: Down/Off 2023-08-16T02:22:17.812 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:22:17.837 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is off 2023-08-16T02:22:17.856 INFO:teuthology.orchestra.console.smithi148:Power off 2023-08-16T02:22:17.857 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-16T02:22:17.880 DEBUG:teuthology.orchestra.console.smithi148:power off output: Chassis Power Control: Down/Off 2023-08-16T02:22:17.881 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:22:17.907 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2023-08-16T02:22:17.939 INFO:teuthology.orchestra.console.smithi081:Power off completed 2023-08-16T02:22:18.040 INFO:teuthology.orchestra.console.smithi081:Power on 2023-08-16T02:22:18.041 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-08-16T02:22:18.064 DEBUG:teuthology.orchestra.console.smithi081:power on output: Chassis Power Control: Up/On 2023-08-16T02:22:18.065 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:22:18.088 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is off 2023-08-16T02:22:21.776 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:22:21.909 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:22:22.089 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:22:22.888 DEBUG:teuthology.orchestra.console.smithi077:check power output: 2023-08-16T02:22:23.019 DEBUG:teuthology.orchestra.console.smithi148:check power output: 2023-08-16T02:22:23.200 DEBUG:teuthology.orchestra.console.smithi081:check power output: 2023-08-16T02:22:26.889 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:22:27.021 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:22:27.201 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:22:27.999 DEBUG:teuthology.orchestra.console.smithi077:check power output: 2023-08-16T02:22:28.132 DEBUG:teuthology.orchestra.console.smithi148:check power output: 2023-08-16T02:22:28.311 DEBUG:teuthology.orchestra.console.smithi081:check power output: 2023-08-16T02:22:32.004 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:22:32.133 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:22:32.243 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is off 2023-08-16T02:22:32.313 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:22:32.345 INFO:teuthology.orchestra.console.smithi148:Power off completed 2023-08-16T02:22:32.447 INFO:teuthology.orchestra.console.smithi148:Power on 2023-08-16T02:22:32.448 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-08-16T02:22:33.114 DEBUG:teuthology.orchestra.console.smithi077:check power output: 2023-08-16T02:22:33.424 DEBUG:teuthology.orchestra.console.smithi081:check power output: 2023-08-16T02:22:37.116 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:22:37.425 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:22:37.477 DEBUG:teuthology.orchestra.console.smithi148:power on output: Chassis Power Control: Up/On 2023-08-16T02:22:37.478 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:22:38.228 DEBUG:teuthology.orchestra.console.smithi077:check power output: 2023-08-16T02:22:38.488 DEBUG:teuthology.orchestra.console.smithi148:check power output: 2023-08-16T02:22:38.537 DEBUG:teuthology.orchestra.console.smithi081:check power output: 2023-08-16T02:22:42.230 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:22:42.490 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:22:42.538 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:22:43.341 DEBUG:teuthology.orchestra.console.smithi077:check power output: 2023-08-16T02:22:43.601 DEBUG:teuthology.orchestra.console.smithi148:check power output: 2023-08-16T02:22:43.651 DEBUG:teuthology.orchestra.console.smithi081:check power output: 2023-08-16T02:22:47.344 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:22:47.602 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:22:47.653 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:22:47.715 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2023-08-16T02:22:47.816 INFO:teuthology.orchestra.console.smithi148:Power on completed 2023-08-16T02:22:47.917 INFO:teuthology.provision.fog.smithi148:Waiting for deploy to finish 2023-08-16T02:22:47.918 DEBUG:teuthology.provision.fog.smithi148:FOG request: GET /task/active None 2023-08-16T02:22:48.457 DEBUG:teuthology.orchestra.console.smithi077:check power output: 2023-08-16T02:22:48.764 DEBUG:teuthology.orchestra.console.smithi081:check power output: 2023-08-16T02:22:52.458 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:22:52.569 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off 2023-08-16T02:22:52.670 INFO:teuthology.orchestra.console.smithi077:Power off completed 2023-08-16T02:22:52.766 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:22:52.776 INFO:teuthology.orchestra.console.smithi077:Power on 2023-08-16T02:22:52.777 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-08-16T02:22:52.799 DEBUG:teuthology.orchestra.console.smithi077:power on output: Chassis Power Control: Up/On 2023-08-16T02:22:52.800 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:22:52.821 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off 2023-08-16T02:22:53.878 DEBUG:teuthology.orchestra.console.smithi081:check power output: 2023-08-16T02:22:56.825 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:22:56.935 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off 2023-08-16T02:22:57.879 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:22:58.990 DEBUG:teuthology.orchestra.console.smithi081:check power output: 2023-08-16T02:23:00.939 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:23:02.049 DEBUG:teuthology.orchestra.console.smithi077:check power output: 2023-08-16T02:23:02.980 DEBUG:teuthology.provision.fog.smithi148:FOG request: GET /task/active None 2023-08-16T02:23:07.985 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:23:07.992 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:23:09.095 DEBUG:teuthology.orchestra.console.smithi081:check power output: 2023-08-16T02:23:09.102 DEBUG:teuthology.orchestra.console.smithi077:check power output: 2023-08-16T02:23:13.100 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:23:13.108 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:23:14.210 DEBUG:teuthology.orchestra.console.smithi081:check power output: 2023-08-16T02:23:14.218 DEBUG:teuthology.orchestra.console.smithi077:check power output: 2023-08-16T02:23:18.212 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:23:18.219 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:23:19.322 DEBUG:teuthology.orchestra.console.smithi081:check power output: 2023-08-16T02:23:19.328 DEBUG:teuthology.orchestra.console.smithi077:check power output: 2023-08-16T02:23:23.068 DEBUG:teuthology.provision.fog.smithi148:FOG request: GET /task/active None 2023-08-16T02:23:28.073 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:23:28.082 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:23:29.184 DEBUG:teuthology.orchestra.console.smithi081:check power output: 2023-08-16T02:23:29.192 DEBUG:teuthology.orchestra.console.smithi077:check power output: 2023-08-16T02:23:33.188 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:23:33.196 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:23:34.299 DEBUG:teuthology.orchestra.console.smithi081:check power output: 2023-08-16T02:23:34.307 DEBUG:teuthology.orchestra.console.smithi077:check power output: 2023-08-16T02:23:38.304 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:23:38.313 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:23:38.426 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2023-08-16T02:23:38.528 INFO:teuthology.orchestra.console.smithi077:Power on completed 2023-08-16T02:23:38.629 INFO:teuthology.provision.fog.smithi077:Waiting for deploy to finish 2023-08-16T02:23:38.629 DEBUG:teuthology.provision.fog.smithi077:FOG request: GET /task/active None 2023-08-16T02:23:39.415 DEBUG:teuthology.orchestra.console.smithi081:check power output: 2023-08-16T02:23:39.415 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds 2023-08-16T02:23:39.416 DEBUG:teuthology.provision.fog.smithi081:Canceling deploy task with ID None 2023-08-16T02:23:39.416 DEBUG:teuthology.provision.fog.smithi081:FOG request: DELETE /task/cancel {"id": None} 2023-08-16T02:23:44.424 DEBUG:teuthology.provision.fog.smithi148:FOG request: GET /task/active None 2023-08-16T02:23:49.484 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/dispatcher/supervisor.py", line 228, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi081 2023-08-16T02:23:49.485 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-16T02:23:53.956 DEBUG:teuthology.provision.fog.smithi077:FOG request: GET /task/active None 2023-08-16T02:23:54.025 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-16T02:23:54.124 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=d5acf3f38f90496092e26337b2182b91 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/dispatcher/supervisor.py", line 228, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi081 2023-08-16T02:23:54.127 INFO:teuthology.nuke:Checking targets against current locks 2023-08-16T02:23:54.183 DEBUG:teuthology.nuke:shortname: smithi077 2023-08-16T02:23:54.183 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-16T02:23:54.210 DEBUG:teuthology.nuke:shortname: smithi081 2023-08-16T02:23:54.210 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-16T02:23:54.213 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-15_18:58:56-rados-wip-yuri3-testing-2023-08-15-0955-distro-default-smithi/7369284', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-16 02:21:11.894276', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0='} 2023-08-16T02:23:54.232 DEBUG:teuthology.nuke:shortname: smithi148 2023-08-16T02:23:54.232 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-16T02:23:54.235 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-15_18:58:56-rados-wip-yuri3-testing-2023-08-15-0955-distro-default-smithi/7369284', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-16 02:21:11.896353', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0='} 2023-08-16T02:23:54.261 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-15_18:58:56-rados-wip-yuri3-testing-2023-08-15-0955-distro-default-smithi/7369284', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-16 02:21:11.898100', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0='} 2023-08-16T02:23:54.269 INFO:teuthology.orchestra.console.smithi077:Power off 2023-08-16T02:23:54.270 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-16T02:23:54.292 DEBUG:teuthology.orchestra.console.smithi077:power off output: Chassis Power Control: Down/Off 2023-08-16T02:23:54.293 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:23:54.309 INFO:teuthology.orchestra.console.smithi081:Power off 2023-08-16T02:23:54.310 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-16T02:23:54.327 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2023-08-16T02:23:54.338 DEBUG:teuthology.orchestra.console.smithi081:power off output: Chassis Power Control: Down/Off 2023-08-16T02:23:54.339 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:23:54.356 INFO:teuthology.orchestra.console.smithi148:Power off 2023-08-16T02:23:54.357 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-16T02:23:54.368 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on 2023-08-16T02:23:54.380 DEBUG:teuthology.orchestra.console.smithi148:power off output: Chassis Power Control: Down/Off 2023-08-16T02:23:54.380 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:23:54.406 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2023-08-16T02:23:57.104 WARNING:teuthology.provision.fog.smithi077:[Errno None] Unable to connect to port 22 on 172.21.15.77 2023-08-16T02:23:58.329 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:23:58.370 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:23:58.408 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:23:59.441 DEBUG:teuthology.orchestra.console.smithi077:check power output: 2023-08-16T02:23:59.480 DEBUG:teuthology.orchestra.console.smithi081:check power output: 2023-08-16T02:23:59.519 DEBUG:teuthology.orchestra.console.smithi148:check power output: 2023-08-16T02:24:03.110 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-16T02:24:03.443 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:24:03.482 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:24:03.521 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-16T02:24:03.634 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is off 2023-08-16T02:24:03.735 INFO:teuthology.orchestra.console.smithi148:Power off completed 2023-08-16T02:24:13.845 DEBUG:teuthology.provision.fog.smithi148:FOG request: GET /task/active None 2023-08-16T02:24:13.846 WARNING:teuthology.provision.fog.smithi077:[Errno None] Unable to connect to port 22 on 172.21.15.77 2023-08-16T02:24:18.852 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off 2023-08-16T02:24:18.853 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is off 2023-08-16T02:24:23.884 INFO:teuthology.orchestra.console.smithi077:Power off completed 2023-08-16T02:24:23.885 INFO:teuthology.orchestra.console.smithi081:Power off completed 2023-08-16T02:24:23.886 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-16T02:24:28.894 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-16T02:24:33.898 WARNING:teuthology.provision.fog.smithi077:[Errno None] Unable to connect to port 22 on 172.21.15.77 2023-08-16T02:24:43.909 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-16T02:24:48.917 WARNING:teuthology.provision.fog.smithi148:[Errno None] Unable to connect to port 22 on 172.21.15.148 2023-08-16T02:24:48.919 INFO:teuthology.lock.ops:unlocked: smithi148.front.sepia.ceph.com 2023-08-16T02:24:53.945 WARNING:teuthology.provision.fog.smithi077:[Errno None] Unable to connect to port 22 on 172.21.15.77 2023-08-16T02:24:58.955 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-16T02:25:03.961 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-16T02:25:08.966 WARNING:teuthology.provision.fog.smithi148:[Errno None] Unable to connect to port 22 on 172.21.15.148 2023-08-16T02:25:08.968 INFO:teuthology.lock.ops:unlocked: smithi077.front.sepia.ceph.com 2023-08-16T02:25:08.990 INFO:teuthology.lock.ops:unlocked: smithi081.front.sepia.ceph.com 2023-08-16T02:25:13.996 WARNING:teuthology.provision.fog.smithi077:[Errno None] Unable to connect to port 22 on 172.21.15.77 2023-08-16T02:25:19.161 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-16T02:25:24.167 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-16T02:25:29.180 WARNING:teuthology.provision.fog.smithi148:[Errno None] Unable to connect to port 22 on 172.21.15.148 2023-08-16T02:25:34.293 WARNING:teuthology.provision.fog.smithi077:[Errno None] Unable to connect to port 22 on 172.21.15.77 2023-08-16T02:25:34.412 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/dispatcher/supervisor.py", line 58, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/dispatcher/supervisor.py", line 228, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi081