2023-07-15T22:25:57.995 INFO:root:teuthology version: 0.0.1.dev147+gff9991b 2023-07-15T22:25:57.997 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-15T22:25:57.997 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-15T22:25:58.205 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1de7944820>, signals=[15]) 2023-07-15T22:25:58.207 INFO:teuthology.lock.ops:Start node 'smithi162.front.sepia.ceph.com' reimaging 2023-07-15T22:25:58.207 INFO:teuthology.lock.ops:Updating [smithi162.front.sepia.ceph.com]: reset os type and version on server 2023-07-15T22:25:58.208 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2023-07-15T22:25:58.235 INFO:teuthology.lock.ops:Start node 'smithi184.front.sepia.ceph.com' reimaging 2023-07-15T22:25:58.236 INFO:teuthology.lock.ops:Updating [smithi184.front.sepia.ceph.com]: reset os type and version on server 2023-07-15T22:25:58.236 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2023-07-15T22:25:58.238 DEBUG:teuthology.provision.fog.smithi162:FOG request: GET /host {"name": "smithi162"} 2023-07-15T22:26:02.276 INFO:teuthology.lock.ops:Start node 'smithi190.front.sepia.ceph.com' reimaging 2023-07-15T22:26:02.277 INFO:teuthology.lock.ops:Updating [smithi190.front.sepia.ceph.com]: reset os type and version on server 2023-07-15T22:26:02.277 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2023-07-15T22:26:02.278 DEBUG:teuthology.provision.fog.smithi184:FOG request: GET /host {"name": "smithi184"} 2023-07-15T22:26:02.301 DEBUG:teuthology.provision.fog.smithi190:FOG request: GET /host {"name": "smithi190"} 2023-07-15T22:26:02.345 DEBUG:teuthology.provision.fog.smithi162:FOG request: GET /image {"name": "smithi_rhel_8.6"} 2023-07-15T22:26:20.901 DEBUG:teuthology.provision.fog.smithi190:FOG request: GET /image {"name": "smithi_rhel_8.6"} 2023-07-15T22:26:25.906 DEBUG:teuthology.provision.fog.smithi162:Requesting image smithi_rhel_8.6 (ID 5) 2023-07-15T22:26:25.907 DEBUG:teuthology.provision.fog.smithi162:FOG request: PUT /host/267 {"imageID": 5} 2023-07-15T22:26:25.908 DEBUG:teuthology.provision.fog.smithi184:FOG request: GET /image {"name": "smithi_rhel_8.6"} 2023-07-15T22:26:40.962 DEBUG:teuthology.provision.fog.smithi184:Requesting image smithi_rhel_8.6 (ID 5) 2023-07-15T22:26:40.962 DEBUG:teuthology.provision.fog.smithi184:FOG request: PUT /host/289 {"imageID": 5} 2023-07-15T22:26:45.970 INFO:teuthology.provision.fog.smithi162:Scheduling deploy of rhel 8.6 2023-07-15T22:26:45.971 DEBUG:teuthology.provision.fog.smithi162:FOG request: GET /task/active None 2023-07-15T22:26:45.972 DEBUG:teuthology.provision.fog.smithi190:Requesting image smithi_rhel_8.6 (ID 5) 2023-07-15T22:26:45.972 DEBUG:teuthology.provision.fog.smithi190:FOG request: PUT /host/295 {"imageID": 5} 2023-07-15T22:26:46.042 INFO:teuthology.provision.fog.smithi184:Scheduling deploy of rhel 8.6 2023-07-15T22:26:46.043 DEBUG:teuthology.provision.fog.smithi184:FOG request: GET /task/active None 2023-07-15T22:26:46.044 INFO:teuthology.provision.fog.smithi190:Scheduling deploy of rhel 8.6 2023-07-15T22:26:46.044 DEBUG:teuthology.provision.fog.smithi190:FOG request: GET /task/active None 2023-07-15T22:26:46.055 DEBUG:teuthology.provision.fog.smithi162:FOG request: GET /tasktype {"name": "deploy"} 2023-07-15T22:26:46.088 DEBUG:teuthology.provision.fog.smithi162:FOG request: POST /host/267/task {"taskTypeID": 1} 2023-07-15T22:26:46.092 DEBUG:teuthology.provision.fog.smithi190:FOG request: GET /tasktype {"name": "deploy"} 2023-07-15T22:26:46.108 DEBUG:teuthology.provision.fog.smithi184:Canceling deploy task with ID 389085 2023-07-15T22:26:46.108 DEBUG:teuthology.provision.fog.smithi184:FOG request: DELETE /task/cancel {"id": 389085} 2023-07-15T22:26:46.125 DEBUG:teuthology.provision.fog.smithi190:FOG request: POST /host/295/task {"taskTypeID": 1} 2023-07-15T22:26:46.153 DEBUG:teuthology.provision.fog.smithi162:FOG request: GET /task/active None 2023-07-15T22:26:46.161 DEBUG:teuthology.provision.fog.smithi184:FOG request: GET /tasktype {"name": "deploy"} 2023-07-15T22:26:46.205 DEBUG:teuthology.provision.fog.smithi184:FOG request: POST /host/289/task {"taskTypeID": 1} 2023-07-15T22:26:48.000 ERROR:teuthology.provision.fog.smithi190:500: { "error": "Host is already a member of an active task" } 2023-07-15T22:26:48.003 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/provision/fog.py", line 79, in create task_id = self.schedule_deploy_task(host_id) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/provision/fog.py", line 211, in schedule_deploy_task resp = self.do_request( File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/provision/fog.py", line 125, in do_request resp.raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/virtualenv/lib/python3.8/site-packages/requests/models.py", line 960, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://fog.front.sepia.ceph.com/fog/host/295/task 2023-07-15T22:26:48.005 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-15T22:26:48.006 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-15T22:26:48.044 INFO:teuthology.orchestra.console:Power off smithi162 2023-07-15T22:26:48.045 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-15T22:26:48.060 DEBUG:teuthology.provision.fog.smithi184:FOG request: GET /task/active None 2023-07-15T22:26:48.072 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T22:26:48.153 INFO:teuthology.nuke:Checking targets against current locks 2023-07-15T22:26:48.169 INFO:teuthology.orchestra.console:Power off smithi184 2023-07-15T22:26:48.170 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-15T22:26:48.212 INFO:teuthology.nuke:Not nuking smithi184.front.sepia.ceph.com because description doesn't match: yuriw-2023-07-14_23:49:32-rbd-reef-distro-default-smithi != /home/teuthworker/archive/yuriw-2023-07-14_23:46:34-rados-reef-distro-default-smithi/7339670 2023-07-15T22:26:48.214 DEBUG:teuthology.nuke:shortname: smithi162 2023-07-15T22:26:48.215 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-15T22:26:48.242 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-14_23:49:32-rbd-reef-distro-default-smithi/7339669', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-15 22:25:57.090432', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKOs9GV3dzbDhHpGEHpXSpDq+YHLCvIas63bx79CVNsfuX/AmjGa8cTh+TxbYGkD9bRhBIXTPnnY1JSGKoM7dO8='} 2023-07-15T22:26:48.267 DEBUG:teuthology.nuke:shortname: smithi190 2023-07-15T22:26:48.267 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-15T22:26:48.296 INFO:teuthology.orchestra.console:Power off smithi162 2023-07-15T22:26:48.297 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-15T22:26:48.308 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-14_23:49:32-rbd-reef-distro-default-smithi/7339671', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-15 22:25:57.952398', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2iyVy/2WvEKSID0qayHJEsGV0mXnlTtQRzRUs6SBFV6Y6uQPR21A0ZWzekae1HbzJJb/5shc82dKzlfNnxHZpBAQju3W1CpUjVjzYfPDDdenZA0GngZEo121VYMi6rca5uCgVDIODcPJjzzXwxeBkZ1qQHErJ/kErJILoIozCnrjBQ2YNzzI4q5EiId8c1QOshFFmhkEROQGbZi3Tn7aUuXqnSZGNi7N9ffmzkMhkKgkoayBDrEKCSv8fgGw+G9rp05cxnEfpsfsFIEQCgz0ysBMY1XMgM8C6YHN+bJieaIsYBlNkAytS3cHbza37FN8KEVhMvu7N+7ML/Efl2G7q+GUsIAfSzzXpVdtl/+bRr7QqOKPkWZNzjBcLsGbf7zEnPwhTh389RA0BpuphKIz77ZFo52fh2IbTHCjljN8y5cMSn5HdBU30ajivo2DlGLoWZPV8OQEs4mD5Y+ls8PHDdRih+rR9vubTNYADr6iSRxkcop8YKmsWTkjE3Fenl5c='} 2023-07-15T22:26:48.320 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T22:26:48.362 INFO:teuthology.orchestra.console:Power off smithi190 2023-07-15T22:26:48.363 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-15T22:26:48.386 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T22:26:52.099 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T22:26:52.343 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T22:26:53.397 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T22:26:57.212 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T22:26:57.454 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T22:26:57.509 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T22:27:02.323 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T22:27:02.566 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T22:27:02.620 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T22:27:03.207 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T22:27:07.434 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T22:27:07.678 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T22:27:07.732 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T22:27:07.942 INFO:teuthology.orchestra.console:Power off for smithi190 completed 2023-07-15T22:27:08.498 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T22:27:08.588 INFO:teuthology.lock.ops:unlocked: smithi190.front.sepia.ceph.com 2023-07-15T22:27:12.544 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T22:27:12.755 INFO:teuthology.orchestra.console:Power off for smithi162 completed 2023-07-15T22:27:12.789 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T22:27:12.857 INFO:teuthology.orchestra.console:Power on smithi162 2023-07-15T22:27:12.857 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-07-15T22:27:12.879 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T22:27:13.001 INFO:teuthology.orchestra.console:Power off for smithi162 completed 2023-07-15T22:27:18.107 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T22:27:18.117 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-15T22:27:23.185 INFO:teuthology.lock.ops:unlocked: smithi162.front.sepia.ceph.com 2023-07-15T22:27:23.187 ERROR:teuthology.nuke:Could not nuke the following targets: targets: smithi184.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMO/Yjiv5irhhK/UhdNiug/QDChQgk4BkjzdZqF11oqW9Y/Q75nhJU0gXPLNVKZioMMtW0RcsH3V64siAG/2cLw= 2023-07-15T22:27:28.193 INFO:teuthology.orchestra.console:Power on for smithi162 completed 2023-07-15T22:27:28.295 INFO:teuthology.provision.fog.smithi162:Waiting for deploy to finish 2023-07-15T22:27:28.295 DEBUG:teuthology.provision.fog.smithi162:FOG request: GET /task/active None 2023-07-15T22:27:28.304 INFO:teuthology.orchestra.console:Power off for smithi184 completed 2023-07-15T22:27:32.178 INFO:teuthology.orchestra.console:Power on smithi184 2023-07-15T22:27:32.178 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-07-15T22:27:32.414 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/dispatcher/supervisor.py", line 57, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/provision/fog.py", line 79, in create task_id = self.schedule_deploy_task(host_id) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/provision/fog.py", line 211, in schedule_deploy_task resp = self.do_request( File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/provision/fog.py", line 125, in do_request resp.raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/virtualenv/lib/python3.8/site-packages/requests/models.py", line 960, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://fog.front.sepia.ceph.com/fog/host/295/task