2024-03-09T21:39:30.020 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-09T21:39:30.020 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-09T21:39:30.163 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7425a7dc10>, signals=[15]) 2024-03-09T21:39:30.163 INFO:teuthology.lock.ops:Start node 'smithi059.front.sepia.ceph.com' reimaging 2024-03-09T21:39:30.163 INFO:teuthology.lock.ops:Updating [smithi059.front.sepia.ceph.com]: reset os type and version on server 2024-03-09T21:39:30.164 INFO:teuthology.lock.ops:Updating smithi059.front.sepia.ceph.com on lock server 2024-03-09T21:39:30.184 INFO:teuthology.lock.ops:Start node 'smithi123.front.sepia.ceph.com' reimaging 2024-03-09T21:39:30.185 INFO:teuthology.lock.ops:Updating [smithi123.front.sepia.ceph.com]: reset os type and version on server 2024-03-09T21:39:30.185 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server 2024-03-09T21:39:30.297 DEBUG:teuthology.provision.fog.smithi059:Requesting image smithi_centos_9.stream (ID 10) 2024-03-09T21:39:30.319 DEBUG:teuthology.provision.fog.smithi123:Requesting image smithi_centos_9.stream (ID 10) 2024-03-09T21:39:30.338 INFO:teuthology.provision.fog.smithi059:Scheduling deploy of centos 9.stream 2024-03-09T21:39:30.364 INFO:teuthology.provision.fog.smithi123:Scheduling deploy of centos 9.stream 2024-03-09T21:39:30.557 INFO:teuthology.orchestra.console.smithi059:Power off 2024-03-09T21:39:30.557 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-09T21:39:30.588 DEBUG:teuthology.orchestra.console.smithi059:power off output: Chassis Power Control: Down/Off 2024-03-09T21:39:30.589 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T21:39:30.602 INFO:teuthology.orchestra.console.smithi123:Power off 2024-03-09T21:39:30.602 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-09T21:39:30.611 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is on 2024-03-09T21:39:30.621 DEBUG:teuthology.orchestra.console.smithi123:power off output: Chassis Power Control: Down/Off 2024-03-09T21:39:30.622 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T21:39:30.643 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is off 2024-03-09T21:39:30.744 INFO:teuthology.orchestra.console.smithi123:Power off completed 2024-03-09T21:39:30.844 INFO:teuthology.orchestra.console.smithi123:Power on 2024-03-09T21:39:30.844 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-09T21:39:30.865 DEBUG:teuthology.orchestra.console.smithi123:power on output: Chassis Power Control: Up/On 2024-03-09T21:39:30.866 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T21:39:30.886 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is off 2024-03-09T21:39:34.615 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T21:39:34.723 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is off 2024-03-09T21:39:34.824 INFO:teuthology.orchestra.console.smithi059:Power off completed 2024-03-09T21:39:34.887 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T21:39:34.925 INFO:teuthology.orchestra.console.smithi059:Power on 2024-03-09T21:39:34.925 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-09T21:39:34.946 DEBUG:teuthology.orchestra.console.smithi059:power on output: Chassis Power Control: Up/On 2024-03-09T21:39:34.946 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T21:39:34.967 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is off 2024-03-09T21:39:34.996 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2024-03-09T21:39:35.097 INFO:teuthology.orchestra.console.smithi123:Power on completed 2024-03-09T21:39:35.197 INFO:teuthology.provision.fog.smithi123:Waiting for deploy to finish 2024-03-09T21:39:38.970 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T21:39:39.079 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is off 2024-03-09T21:39:43.082 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T21:39:43.191 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is off 2024-03-09T21:39:47.194 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T21:39:47.303 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is off 2024-03-09T21:39:51.305 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T21:39:51.414 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is off 2024-03-09T21:39:55.418 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T21:39:55.528 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is off 2024-03-09T21:39:59.530 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T21:39:59.642 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is off 2024-03-09T21:40:03.645 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T21:40:03.755 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is off 2024-03-09T21:40:07.756 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T21:40:07.865 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is off 2024-03-09T21:40:11.870 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T21:40:11.979 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is off 2024-03-09T21:40:15.984 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T21:40:16.093 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is off 2024-03-09T21:40:20.097 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T21:40:20.205 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is off 2024-03-09T21:40:24.207 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T21:40:24.316 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is off 2024-03-09T21:40:28.318 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T21:40:28.427 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is off 2024-03-09T21:40:32.430 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T21:40:32.538 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is off 2024-03-09T21:40:32.539 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds 2024-03-09T21:40:32.639 DEBUG:teuthology.provision.fog.smithi059:Canceling deploy task with ID 844387 2024-03-09T21:40:32.790 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi059 2024-03-09T21:40:32.791 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-09T21:40:33.091 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=df4688fb3e5a46adbfd2b489df71e280 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi059 2024-03-09T21:40:33.093 INFO:teuthology.nuke:Checking targets against current locks 2024-03-09T21:40:33.157 DEBUG:teuthology.nuke:shortname: smithi059 2024-03-09T21:40:33.158 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-09T21:40:33.178 DEBUG:teuthology.nuke:shortname: smithi123 2024-03-09T21:40:33.178 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-09T21:40:33.181 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-09_19:05:46-fs:functional-wip-batrick-testing-20240308.191858-distro-default-smithi/7588962', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-09 21:39:28.987540', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAa76OSpDX65MFvaSj0oUpHq1XjZyBJiVj40/8szGXS+ziD5IVxQiDBZN8KrUFooFzEOZmTtSepHY/btVcMwN1k='} 2024-03-09T21:40:33.203 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-09_19:05:46-fs:functional-wip-batrick-testing-20240308.191858-distro-default-smithi/7588962', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-09 21:39:28.988703', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFCA05CUEKYkDKmPOSwp+a3DCzbXRAb/lIrBREABP9AGiA7ZApPDhHG+IR+4JpCy1qmvCqwBdSlDY0WLitGolRs='} 2024-03-09T21:40:33.229 INFO:teuthology.orchestra.console.smithi059:Power off 2024-03-09T21:40:33.229 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-09T21:40:33.251 DEBUG:teuthology.orchestra.console.smithi059:power off output: Chassis Power Control: Down/Off 2024-03-09T21:40:33.252 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T21:40:33.269 INFO:teuthology.orchestra.console.smithi123:Power off 2024-03-09T21:40:33.270 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-09T21:40:33.280 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is off 2024-03-09T21:40:33.292 DEBUG:teuthology.orchestra.console.smithi123:power off output: Chassis Power Control: Down/Off 2024-03-09T21:40:33.292 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T21:40:33.312 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2024-03-09T21:40:33.382 INFO:teuthology.orchestra.console.smithi059:Power off completed 2024-03-09T21:40:33.536 INFO:teuthology.lock.ops:unlocked: smithi059.front.sepia.ceph.com 2024-03-09T21:40:37.314 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T21:40:37.424 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is off 2024-03-09T21:40:37.525 INFO:teuthology.orchestra.console.smithi123:Power off completed 2024-03-09T21:40:37.678 INFO:teuthology.lock.ops:unlocked: smithi123.front.sepia.ceph.com 2024-03-09T21:40:37.967 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 58, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi059