2023-11-06T19:47:00.217 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-06T19:47:00.218 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-06T19:47:00.325 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd912e29af0>, signals=[15]) 2023-11-06T19:47:00.326 INFO:teuthology.lock.ops:Start node 'smithi113.front.sepia.ceph.com' reimaging 2023-11-06T19:47:00.327 INFO:teuthology.lock.ops:Updating [smithi113.front.sepia.ceph.com]: reset os type and version on server 2023-11-06T19:47:00.327 INFO:teuthology.lock.ops:Updating smithi113.front.sepia.ceph.com on lock server 2023-11-06T19:47:00.343 DEBUG:teuthology.provision.fog.smithi113:FOG request: GET /host {"name": "smithi113"} 2023-11-06T19:47:00.418 DEBUG:teuthology.provision.fog.smithi113:FOG request: GET /image {"name": "smithi_ubuntu_20.04"} 2023-11-06T19:47:00.462 DEBUG:teuthology.provision.fog.smithi113:Requesting image smithi_ubuntu_20.04 (ID 2) 2023-11-06T19:47:00.462 DEBUG:teuthology.provision.fog.smithi113:FOG request: PUT /host/218 {"imageID": 2} 2023-11-06T19:47:00.517 INFO:teuthology.provision.fog.smithi113:Scheduling deploy of ubuntu 20.04 2023-11-06T19:47:00.517 DEBUG:teuthology.provision.fog.smithi113:FOG request: GET /task/active None 2023-11-06T19:47:00.583 DEBUG:teuthology.provision.fog.smithi113:FOG request: GET /tasktype {"name": "deploy"} 2023-11-06T19:47:00.628 DEBUG:teuthology.provision.fog.smithi113:FOG request: POST /host/218/task {"taskTypeID": 1} 2023-11-06T19:47:00.688 DEBUG:teuthology.provision.fog.smithi113:FOG request: GET /task/active None 2023-11-06T19:47:00.773 INFO:teuthology.orchestra.console.smithi113:Power off 2023-11-06T19:47:00.774 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-06T19:47:20.839 DEBUG:teuthology.orchestra.console.smithi113:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-11-06T19:47:20.839 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-06T19:47:40.910 DEBUG:teuthology.orchestra.console.smithi113:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-11-06T19:47:40.911 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-06T19:48:00.982 DEBUG:teuthology.orchestra.console.smithi113:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-11-06T19:48:00.983 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:48:01.992 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:48:05.994 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:48:07.102 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:48:11.107 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:48:12.216 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:48:16.220 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:48:17.329 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:48:21.330 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:48:22.438 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:48:26.442 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:48:27.550 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:48:31.554 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:48:32.662 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:48:36.664 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:48:37.772 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:48:41.774 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:48:42.883 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:48:46.886 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:48:47.994 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:48:51.998 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:48:53.108 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:48:57.110 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:48:58.221 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:49:02.223 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:49:03.332 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:49:07.334 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:49:08.446 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:49:12.450 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:49:13.558 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:49:13.559 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds 2023-11-06T19:49:13.560 ERROR:teuthology.orchestra.console.smithi113:Failed to power off 2023-11-06T19:49:13.560 INFO:teuthology.orchestra.console.smithi113:Power on 2023-11-06T19:49:13.560 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-11-06T19:49:33.628 DEBUG:teuthology.orchestra.console.smithi113:power on output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-11-06T19:49:33.629 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-11-06T19:49:53.697 DEBUG:teuthology.orchestra.console.smithi113:power on output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-11-06T19:49:53.698 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-11-06T19:50:13.768 DEBUG:teuthology.orchestra.console.smithi113:power on output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-11-06T19:50:13.769 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:50:14.780 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:50:18.782 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:50:19.891 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:50:23.894 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:50:25.003 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:50:29.006 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:50:30.114 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:50:34.115 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:50:35.224 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:50:39.226 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:50:40.335 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:50:44.338 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:50:45.446 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:50:49.450 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:50:50.558 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:50:54.562 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:50:55.671 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:50:59.674 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:51:00.784 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:51:04.786 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:51:05.896 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:51:09.900 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:51:11.009 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:51:15.014 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:51:16.124 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:51:20.129 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:51:21.238 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:51:25.242 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:51:26.352 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:51:26.352 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds 2023-11-06T19:51:26.353 DEBUG:teuthology.provision.fog.smithi113:Canceling deploy task with ID 589422 2023-11-06T19:51:26.353 DEBUG:teuthology.provision.fog.smithi113:FOG request: DELETE /task/cancel {"id": 589422} 2023-11-06T19:51:26.419 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi113 2023-11-06T19:51:26.420 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-06T19:51:26.564 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=7ea37a52b4e74c2b9feaf9ba9b137681 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi113 2023-11-06T19:51:26.569 INFO:teuthology.nuke:Checking targets against current locks 2023-11-06T19:51:26.641 DEBUG:teuthology.nuke:shortname: smithi113 2023-11-06T19:51:26.641 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-06T19:51:26.667 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi113.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-06_19:45:43-upgrade:quincy-x-reef-release-distro-default-smithi/7450257', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-06 19:46:59.455790', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAJ0kcEGTlQ4A59UuFWiVknInZGHNb7vPXKS6mh1vNuSzpoPJgVRbNoxQXpBhFcAkMSEttf+6KPvnUhpt/5kX1SMFngL0egU1cqh6ENnbEX3KTXySW5ZxK+NQ5CZFcRmHjYhNHul5zLADAnftE/NlLTJc3ESNTYxxUU1KnhJGF7xTG5cCZQ8QwtC3sl3wSVIoI4YxiQZIWXBvCnXgvY65ePjP2Zsb/S+G+S4GGQR002bFQXOczlBXrBnQvBok26KAujKlsBC/fn+e1cMIMzCcA3MjWgvGaTTBACvCdWF5pQk0q1yVKr31ol5Ncg/MwitULW/VGwrqt0gKCQpDoks4gG9UySiC4uKMiQ8It9oqnN8HpbqaRIWMNRRAHL/HZWC67vBo+30EBT526xbk5xQ18xiIYmmiLY7007wuzFgdSP9PthALrgK3PUttEiGElRHjjHXVVvtVTylA6+Mz7TKVqwiGonQZHThsa09vyMpX7mc5ABXnTctoHWb5GqN6uRts='} 2023-11-06T19:51:26.721 INFO:teuthology.orchestra.console.smithi113:Power off 2023-11-06T19:51:26.721 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-06T19:51:46.793 DEBUG:teuthology.orchestra.console.smithi113:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-11-06T19:51:46.794 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-06T19:52:06.866 DEBUG:teuthology.orchestra.console.smithi113:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-11-06T19:52:06.867 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-06T19:52:26.942 DEBUG:teuthology.orchestra.console.smithi113:power off output: Error: Unable to establish IPMI v2 / RMCP+ session 2023-11-06T19:52:26.943 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:52:27.951 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:52:31.953 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:52:33.063 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:52:37.067 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:52:38.176 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:52:42.181 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:52:43.290 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:52:47.295 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:52:48.405 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:52:52.410 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:52:53.518 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:52:57.522 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:52:58.632 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:53:02.637 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:53:03.751 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:53:07.754 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:53:08.863 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:53:12.866 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:53:13.975 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:53:17.980 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:53:19.089 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:53:23.094 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:53:24.203 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:53:28.208 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:53:29.317 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:53:33.322 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:53:34.430 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:53:38.435 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-06T19:53:39.545 DEBUG:teuthology.orchestra.console.smithi113:check power output: 2023-11-06T19:53:39.546 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds 2023-11-06T19:53:39.546 ERROR:teuthology.orchestra.console.smithi113:Failed to power off 2023-11-06T19:53:39.690 INFO:teuthology.lock.ops:unlocked: smithi113.front.sepia.ceph.com 2023-11-06T19:53:44.714 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 58, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/dispatcher/supervisor.py", line 228, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi113