2023-07-27T00:59:15.543 INFO:root:teuthology version: 0.0.1.dev160+g407880c 2023-07-27T00:59:15.544 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-27T00:59:15.545 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-27T00:59:15.710 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7f71f1b310>, signals=[15]) 2023-07-27T00:59:15.712 INFO:teuthology.lock.ops:Start node 'smithi144.front.sepia.ceph.com' reimaging 2023-07-27T00:59:15.712 INFO:teuthology.lock.ops:Updating [smithi144.front.sepia.ceph.com]: reset os type and version on server 2023-07-27T00:59:15.712 INFO:teuthology.lock.ops:Updating smithi144.front.sepia.ceph.com on lock server 2023-07-27T00:59:15.738 INFO:teuthology.lock.ops:Start node 'smithi192.front.sepia.ceph.com' reimaging 2023-07-27T00:59:15.738 INFO:teuthology.lock.ops:Updating [smithi192.front.sepia.ceph.com]: reset os type and version on server 2023-07-27T00:59:15.738 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2023-07-27T00:59:15.740 DEBUG:teuthology.provision.fog.smithi144:FOG request: GET /host {"name": "smithi144"} 2023-07-27T00:59:21.837 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /host {"name": "smithi192"} 2023-07-27T00:59:22.279 DEBUG:teuthology.provision.fog.smithi144:FOG request: GET /image {"name": "smithi_centos_8.stream"} 2023-07-27T00:59:22.283 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /image {"name": "smithi_centos_8.stream"} 2023-07-27T00:59:22.327 DEBUG:teuthology.provision.fog.smithi144:Requesting image smithi_centos_8.stream (ID 3) 2023-07-27T00:59:22.327 DEBUG:teuthology.provision.fog.smithi144:FOG request: PUT /host/249 {"imageID": 3} 2023-07-27T00:59:22.332 DEBUG:teuthology.provision.fog.smithi192:Requesting image smithi_centos_8.stream (ID 3) 2023-07-27T00:59:22.332 DEBUG:teuthology.provision.fog.smithi192:FOG request: PUT /host/297 {"imageID": 3} 2023-07-27T00:59:22.369 INFO:teuthology.provision.fog.smithi144:Scheduling deploy of centos 8.stream 2023-07-27T00:59:22.369 DEBUG:teuthology.provision.fog.smithi144:FOG request: GET /task/active None 2023-07-27T00:59:22.376 INFO:teuthology.provision.fog.smithi192:Scheduling deploy of centos 8.stream 2023-07-27T00:59:22.377 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-27T00:59:22.424 DEBUG:teuthology.provision.fog.smithi144:FOG request: GET /tasktype {"name": "deploy"} 2023-07-27T00:59:22.427 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /tasktype {"name": "deploy"} 2023-07-27T00:59:22.459 DEBUG:teuthology.provision.fog.smithi192:FOG request: POST /host/297/task {"taskTypeID": 1} 2023-07-27T00:59:22.463 DEBUG:teuthology.provision.fog.smithi144:FOG request: POST /host/249/task {"taskTypeID": 1} 2023-07-27T00:59:22.522 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-27T00:59:22.569 DEBUG:teuthology.provision.fog.smithi144:FOG request: GET /task/active None 2023-07-27T00:59:22.633 INFO:teuthology.orchestra.console:Power off smithi192 2023-07-27T00:59:22.634 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-27T00:59:22.651 INFO:teuthology.orchestra.console:Power off smithi144 2023-07-27T00:59:22.652 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-27T00:59:22.661 DEBUG:teuthology.orchestra.console:power off output: Chassis Power Control: Down/Off 2023-07-27T00:59:22.662 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T00:59:22.675 DEBUG:teuthology.orchestra.console:power off output: Chassis Power Control: Down/Off 2023-07-27T00:59:22.675 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T00:59:22.685 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on 2023-07-27T00:59:22.699 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on 2023-07-27T00:59:26.688 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T00:59:26.701 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T00:59:26.799 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on 2023-07-27T00:59:26.810 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on 2023-07-27T00:59:30.804 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T00:59:30.812 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T00:59:31.914 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T00:59:31.921 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T00:59:35.915 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T00:59:35.923 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T00:59:36.032 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off 2023-07-27T00:59:36.134 INFO:teuthology.orchestra.console:Power off for smithi144 completed 2023-07-27T00:59:36.234 INFO:teuthology.orchestra.console:Power on smithi144 2023-07-27T00:59:36.235 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-07-27T00:59:36.257 DEBUG:teuthology.orchestra.console:power on output: Chassis Power Control: Up/On 2023-07-27T00:59:36.258 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T00:59:36.279 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off 2023-07-27T00:59:37.026 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T00:59:40.284 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T00:59:41.027 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T00:59:41.393 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T00:59:42.136 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T00:59:45.394 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T00:59:46.138 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T00:59:46.505 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T00:59:47.249 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T00:59:50.508 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T00:59:51.250 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T00:59:51.619 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T00:59:52.361 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T00:59:55.620 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T00:59:56.363 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T00:59:56.731 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T00:59:57.474 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:00:00.736 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:00:01.475 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:00:01.846 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:00:02.585 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:00:05.848 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:00:06.587 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:00:06.958 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:00:07.696 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:00:10.959 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:00:11.698 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:00:12.069 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:00:12.809 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:00:16.072 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:00:16.810 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:00:17.182 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:00:17.920 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:00:21.186 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:00:21.921 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:00:22.295 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:00:23.032 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:00:26.299 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:00:27.034 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:00:27.409 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:00:28.144 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:00:31.412 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:00:32.145 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:00:32.255 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off 2023-07-27T01:00:32.356 INFO:teuthology.orchestra.console:Power off for smithi192 completed 2023-07-27T01:00:32.457 INFO:teuthology.orchestra.console:Power on smithi192 2023-07-27T01:00:32.458 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-07-27T01:00:32.478 DEBUG:teuthology.orchestra.console:power on output: Chassis Power Control: Up/On 2023-07-27T01:00:32.479 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:00:32.498 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off 2023-07-27T01:00:32.521 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:00:36.499 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:00:36.522 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:00:37.610 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:00:37.631 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:00:41.611 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:00:41.633 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:00:41.720 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on 2023-07-27T01:00:41.822 INFO:teuthology.orchestra.console:Power on for smithi192 completed 2023-07-27T01:00:41.922 INFO:teuthology.provision.fog.smithi192:Waiting for deploy to finish 2023-07-27T01:00:41.923 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-27T01:00:46.521 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:00:50.524 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:00:51.636 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:00:51.637 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds 2023-07-27T01:00:51.637 DEBUG:teuthology.provision.fog.smithi144:Canceling deploy task with ID 413495 2023-07-27T01:00:51.638 DEBUG:teuthology.provision.fog.smithi144:FOG request: DELETE /task/cancel {"id": 413495} 2023-07-27T01:00:56.643 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi144 2023-07-27T01:00:56.644 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-27T01:00:56.644 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-27T01:01:01.650 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-27T01:01:01.744 INFO:teuthology.nuke:Checking targets against current locks 2023-07-27T01:01:07.525 DEBUG:teuthology.nuke:shortname: smithi144 2023-07-27T01:01:07.526 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-27T01:01:07.551 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi144.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-26_15:54:22-rados-wip-yuri6-testing-2023-07-24-0819-pacific-distro-default-smithi/7353358', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-27 00:59:14.700483', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-27T01:01:07.553 DEBUG:teuthology.nuke:shortname: smithi192 2023-07-27T01:01:07.554 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-27T01:01:07.576 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-26_15:54:22-rados-wip-yuri6-testing-2023-07-24-0819-pacific-distro-default-smithi/7353358', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-27 00:59:14.699014', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-27T01:01:07.601 INFO:teuthology.orchestra.console:Power off smithi144 2023-07-27T01:01:07.601 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-27T01:01:07.626 DEBUG:teuthology.orchestra.console:power off output: Chassis Power Control: Down/Off 2023-07-27T01:01:07.627 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:01:07.643 INFO:teuthology.orchestra.console:Power off smithi192 2023-07-27T01:01:07.643 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-27T01:01:07.654 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on 2023-07-27T01:01:07.666 DEBUG:teuthology.orchestra.console:power off output: Chassis Power Control: Down/Off 2023-07-27T01:01:07.666 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:01:07.687 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on 2023-07-27T01:01:11.658 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:01:11.688 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:01:12.768 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:01:12.798 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:01:16.713 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-27T01:01:21.721 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:01:21.728 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:01:22.831 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:01:22.838 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:01:26.832 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:01:26.839 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:01:27.942 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:01:27.949 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:01:31.944 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:01:31.952 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:01:33.055 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:01:33.061 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:01:36.801 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-27T01:01:41.809 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:01:41.816 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:01:42.006 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off 2023-07-27T01:01:42.107 INFO:teuthology.orchestra.console:Power off for smithi192 completed 2023-07-27T01:01:43.747 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:01:43.801 INFO:teuthology.lock.ops:unlocked: smithi192.front.sepia.ceph.com 2023-07-27T01:01:47.752 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:01:48.864 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:01:52.865 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:01:53.976 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:01:56.879 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-27T01:01:57.978 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:01:59.088 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:02:03.090 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:02:04.200 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:02:08.202 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:02:09.312 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:02:11.940 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-27T01:02:16.945 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:02:18.056 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:02:22.060 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:02:23.169 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:02:27.172 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:02:28.282 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:02:32.013 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-27T01:02:32.284 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:02:32.393 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off 2023-07-27T01:02:32.494 INFO:teuthology.orchestra.console:Power off for smithi144 completed 2023-07-27T01:02:32.650 INFO:teuthology.lock.ops:unlocked: smithi144.front.sepia.ceph.com 2023-07-27T01:02:32.891 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/dispatcher/supervisor.py", line 57, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi144