2023-07-15T07:44:47.581 INFO:root:teuthology version: 0.0.1.dev147+gff9991b 2023-07-15T07:44:47.582 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-15T07:44:47.583 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-15T07:44:47.698 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5ff8216820>, signals=[15]) 2023-07-15T07:44:47.700 INFO:teuthology.lock.ops:Start node 'smithi037.front.sepia.ceph.com' reimaging 2023-07-15T07:44:47.700 INFO:teuthology.lock.ops:Updating [smithi037.front.sepia.ceph.com]: reset os type and version on server 2023-07-15T07:44:47.700 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server 2023-07-15T07:44:47.727 DEBUG:teuthology.provision.fog.smithi037:FOG request: GET /host {"name": "smithi037"} 2023-07-15T07:44:57.918 DEBUG:teuthology.provision.fog.smithi037:FOG request: GET /image {"name": "smithi_rhel_8.6"} 2023-07-15T07:45:02.963 DEBUG:teuthology.provision.fog.smithi037:Requesting image smithi_rhel_8.6 (ID 5) 2023-07-15T07:45:02.963 DEBUG:teuthology.provision.fog.smithi037:FOG request: PUT /host/142 {"imageID": 5} 2023-07-15T07:45:08.019 INFO:teuthology.provision.fog.smithi037:Scheduling deploy of rhel 8.6 2023-07-15T07:45:08.019 DEBUG:teuthology.provision.fog.smithi037:FOG request: GET /task/active None 2023-07-15T07:45:12.189 DEBUG:teuthology.provision.fog.smithi037:FOG request: GET /tasktype {"name": "deploy"} 2023-07-15T07:45:12.230 DEBUG:teuthology.provision.fog.smithi037:FOG request: POST /host/142/task {"taskTypeID": 1} 2023-07-15T07:45:12.284 DEBUG:teuthology.provision.fog.smithi037:FOG request: GET /task/active None 2023-07-15T07:45:12.384 INFO:teuthology.orchestra.console:Power off smithi037 2023-07-15T07:45:12.385 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-15T07:45:12.405 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T07:45:16.430 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T07:45:20.540 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T07:45:25.651 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T07:45:30.761 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T07:45:30.972 INFO:teuthology.orchestra.console:Power off for smithi037 completed 2023-07-15T07:45:31.073 INFO:teuthology.orchestra.console:Power on smithi037 2023-07-15T07:45:31.074 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-07-15T07:45:31.094 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T07:45:35.120 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T07:45:39.232 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T07:45:44.343 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T07:45:49.456 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T07:45:54.568 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T07:45:59.682 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T07:46:04.793 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T07:46:09.907 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T07:46:15.021 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T07:46:20.132 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T07:46:25.244 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T07:46:30.356 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T07:46:35.468 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T07:46:40.584 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T07:46:41.694 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds 2023-07-15T07:46:41.695 DEBUG:teuthology.provision.fog.smithi037:Canceling deploy task with ID 386756 2023-07-15T07:46:41.695 DEBUG:teuthology.provision.fog.smithi037:FOG request: DELETE /task/cancel {"id": 386756} 2023-07-15T07:46:42.438 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/orchestra/console.py", line 243, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi037 2023-07-15T07:46:42.439 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-15T07:46:42.439 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-15T07:46:47.529 INFO:teuthology.nuke:Checking targets against current locks 2023-07-15T07:46:57.593 DEBUG:teuthology.nuke:shortname: smithi037 2023-07-15T07:46:57.594 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-15T07:47:00.474 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi037.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-14_23:39:06-rados-wip-yuri5-testing-2023-07-14-0757-distro-default-smithi/7338787', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-15 07:44:46.802875', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdpJgs3RL55bxLcYLFXKMw5aM0zw1YxObjxi/pf0O4EOG6CNduCBw4OMe44MmWxn0STXCP+hGotTfz7pwbZO1uQmrdStLwjYwuK3V9VOLsSkLmYIDAH8gLPWkTANbb4D020WUTxGUgCiBQPWOns5PRNpwYlsQkH5TmQb110sS+GZ40YxffuKRFJJOZXv2hkapxiph3lsQXCDTceCNNLtU4oQgP5LJbHTDCIOtZCu1+Bxide4z35d0weEzmdOKjKXKXxqXOpG9LzGYsRfSh4AS2vJ1s4F+4VYqS2sbagrMhF+PkcsEWQGPrnzMm4xIcZc+Q3NiWo9NQi4I5KLAauBvE0M95Pa0RMIor98kcLBZ7+pIauCBv3bXSdY02RA6+VgYsiKCzzpEqKy2YlEfr7Cgd2XpeGfh5slxAlPnb87k5m2loisSCmEdHrIGkMey+LQOUYfbLlU619Myk4tC34GpHBsug/ybq/aeK4QZ1GAJDZjxa343UAmDmHs/Yu4t+6mM='} 2023-07-15T07:47:00.530 INFO:teuthology.orchestra.console:Power off smithi037 2023-07-15T07:47:00.531 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-15T07:47:01.039 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T07:47:05.064 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T07:47:10.175 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T07:47:15.288 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T07:47:20.399 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T07:47:20.611 INFO:teuthology.orchestra.console:Power off for smithi037 completed 2023-07-15T07:47:32.971 INFO:teuthology.lock.ops:unlocked: smithi037.front.sepia.ceph.com 2023-07-15T07:47:33.102 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/dispatcher/supervisor.py", line 57, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/orchestra/console.py", line 243, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi037