2023-06-28T15:24:01.822 INFO:root:teuthology version: 0.0.1.dev132+gaa9345a 2023-06-28T15:24:01.823 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-28T15:24:01.824 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-28T15:24:01.971 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4529b353a0>, signals=[15]) 2023-06-28T15:24:01.973 INFO:teuthology.lock.ops:Start node 'smithi038.front.sepia.ceph.com' reimaging 2023-06-28T15:24:01.973 INFO:teuthology.lock.ops:Updating [smithi038.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T15:24:01.973 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2023-06-28T15:24:07.145 INFO:teuthology.lock.ops:Start node 'smithi079.front.sepia.ceph.com' reimaging 2023-06-28T15:24:07.145 INFO:teuthology.lock.ops:Updating [smithi079.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T15:24:07.146 INFO:teuthology.lock.ops:Updating smithi079.front.sepia.ceph.com on lock server 2023-06-28T15:24:07.147 DEBUG:teuthology.provision.fog.smithi038:FOG request: GET /host {"name": "smithi038"} 2023-06-28T15:24:07.702 DEBUG:teuthology.provision.fog.smithi079:FOG request: GET /host {"name": "smithi079"} 2023-06-28T15:24:07.765 DEBUG:teuthology.provision.fog.smithi038:FOG request: GET /image {"name": "smithi_centos_8.stream"} 2023-06-28T15:24:07.794 DEBUG:teuthology.provision.fog.smithi079:FOG request: GET /image {"name": "smithi_centos_8.stream"} 2023-06-28T15:24:07.813 DEBUG:teuthology.provision.fog.smithi038:Requesting image smithi_centos_8.stream (ID 3) 2023-06-28T15:24:07.813 DEBUG:teuthology.provision.fog.smithi038:FOG request: PUT /host/143 {"imageID": 3} 2023-06-28T15:24:07.833 DEBUG:teuthology.provision.fog.smithi079:Requesting image smithi_centos_8.stream (ID 3) 2023-06-28T15:24:07.833 DEBUG:teuthology.provision.fog.smithi079:FOG request: PUT /host/184 {"imageID": 3} 2023-06-28T15:24:07.866 INFO:teuthology.provision.fog.smithi038:Scheduling deploy of centos 8.stream 2023-06-28T15:24:07.866 DEBUG:teuthology.provision.fog.smithi038:FOG request: GET /task/active None 2023-06-28T15:24:07.881 INFO:teuthology.provision.fog.smithi079:Scheduling deploy of centos 8.stream 2023-06-28T15:24:07.882 DEBUG:teuthology.provision.fog.smithi079:FOG request: GET /task/active None 2023-06-28T15:24:07.939 DEBUG:teuthology.provision.fog.smithi038:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T15:24:07.949 DEBUG:teuthology.provision.fog.smithi079:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T15:24:07.980 DEBUG:teuthology.provision.fog.smithi038:FOG request: POST /host/143/task {"taskTypeID": 1} 2023-06-28T15:24:07.989 DEBUG:teuthology.provision.fog.smithi079:FOG request: POST /host/184/task {"taskTypeID": 1} 2023-06-28T15:24:08.036 DEBUG:teuthology.provision.fog.smithi038:FOG request: GET /task/active None 2023-06-28T15:24:08.052 DEBUG:teuthology.provision.fog.smithi079:FOG request: GET /task/active None 2023-06-28T15:24:08.134 INFO:teuthology.orchestra.console:Power off smithi038 2023-06-28T15:24:08.135 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T15:24:08.156 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:24:08.179 INFO:teuthology.orchestra.console:Power off smithi079 2023-06-28T15:24:08.180 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T15:24:08.200 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:24:08.289 INFO:teuthology.orchestra.console:Power off for smithi038 completed 2023-06-28T15:24:08.321 INFO:teuthology.orchestra.console:Power off for smithi079 completed 2023-06-28T15:24:08.391 INFO:teuthology.orchestra.console:Power on smithi038 2023-06-28T15:24:08.392 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T15:24:08.414 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:24:08.424 INFO:teuthology.orchestra.console:Power on smithi079 2023-06-28T15:24:08.424 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T15:24:08.445 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:24:12.440 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:24:12.468 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:24:17.552 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:24:17.578 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:24:22.664 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:24:22.690 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:24:27.775 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:24:27.800 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:24:32.888 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:24:32.911 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:24:38.001 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:24:38.022 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:24:43.112 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:24:43.132 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:24:48.224 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:24:48.243 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:24:53.336 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:24:53.353 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:24:58.448 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:24:58.463 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:25:03.565 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:25:03.574 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:25:08.677 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:25:08.686 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:25:13.788 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:25:13.798 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:25:18.900 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:25:18.910 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:25:20.012 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds 2023-06-28T15:25:20.013 DEBUG:teuthology.provision.fog.smithi038:Canceling deploy task with ID 352024 2023-06-28T15:25:20.014 DEBUG:teuthology.provision.fog.smithi038:FOG request: DELETE /task/cancel {"id": 352024} 2023-06-28T15:25:24.998 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds 2023-06-28T15:25:24.998 DEBUG:teuthology.provision.fog.smithi079:Canceling deploy task with ID 352026 2023-06-28T15:25:24.999 DEBUG:teuthology.provision.fog.smithi079:FOG request: DELETE /task/cancel {"id": 352026} 2023-06-28T15:25:25.180 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/console.py", line 243, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi079 2023-06-28T15:25:25.181 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-28T15:25:25.181 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-28T15:25:30.274 INFO:teuthology.nuke:Checking targets against current locks 2023-06-28T15:25:48.011 DEBUG:teuthology.nuke:shortname: smithi038 2023-06-28T15:25:48.012 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T15:25:48.044 DEBUG:teuthology.nuke:shortname: smithi079 2023-06-28T15:25:48.045 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T15:25:48.048 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-28_14:13:04-rados-quincy_release-distro-default-smithi/7320258', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 15:23:50.998944', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDImdfg0+i8Fer1dAAOFHHnxpTA2e4kVeVFeuv1JbPQZJ5Ih7SDZgiv47Rv6gS0NZ9Kaq06vafqv6YwrMYJI/PJhk643jYee2zBvYEFHsSJJeaa+HFz4dMLsrzhsosivWjXDqCgBFk1AwnjjretvwjjJdNfbsa6R5brdeFwu8saP5EHL5BAbRs4FFSQ292STqvt1WNQgOijY4YlHjm5yAUHtiTweo/flqS7Jr5xUt7AU76Mlp5gCIi3ZWk7WqyVsq/zOS4O+0oXIdcxhPv6FJ64i04dHlABR3dcANTzmjVF+TsR1Yv0nQNvd/wuuskLaQ5Laz9uV7ZGN5Zmgv/8s26GxVb+LBRaCS0vPIiFOVsjhL1rL47u6//m2+tRZ9vI9X6jTWwtqQonaKdFPGD8aiMo6yyasHz6E7gYLP9GzmMnGNMUm+GU4Hauhv8okOm6p8bFCTzPqEG48VYEoS1Tz9818Ir5A6sOIzbYDMlpg/kFSmyEB4tmM3UpCLp2DN47z08='} 2023-06-28T15:25:48.069 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-28_14:13:04-rados-quincy_release-distro-default-smithi/7320258', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 15:23:51.001217', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-06-28T15:25:48.097 INFO:teuthology.orchestra.console:Power off smithi038 2023-06-28T15:25:48.098 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T15:25:48.119 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:25:48.136 INFO:teuthology.orchestra.console:Power off smithi079 2023-06-28T15:25:48.136 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T15:25:48.157 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:25:52.148 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:25:52.180 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:25:56.263 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:25:56.291 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi079.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:25:56.505 INFO:teuthology.orchestra.console:Power off for smithi079 completed 2023-06-28T15:26:06.612 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T15:26:11.684 INFO:teuthology.lock.ops:unlocked: smithi079.front.sepia.ceph.com 2023-06-28T15:26:11.754 INFO:teuthology.orchestra.console:Power off for smithi038 completed 2023-06-28T15:26:21.915 INFO:teuthology.lock.ops:unlocked: smithi038.front.sepia.ceph.com 2023-06-28T15:26:22.184 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 57, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/console.py", line 243, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi079