2024-03-24T21:03:09.552 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-24T21:03:09.552 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T21:03:09.680 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8cac142c10>, signals=[15]) 2024-03-24T21:03:09.680 INFO:teuthology.lock.ops:Start node 'smithi055.front.sepia.ceph.com' reimaging 2024-03-24T21:03:09.681 INFO:teuthology.lock.ops:Updating [smithi055.front.sepia.ceph.com]: reset os type and version on server 2024-03-24T21:03:09.681 INFO:teuthology.lock.ops:Updating smithi055.front.sepia.ceph.com on lock server 2024-03-24T21:03:09.710 INFO:teuthology.lock.ops:Start node 'smithi194.front.sepia.ceph.com' reimaging 2024-03-24T21:03:09.710 INFO:teuthology.lock.ops:Updating [smithi194.front.sepia.ceph.com]: reset os type and version on server 2024-03-24T21:03:09.710 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server 2024-03-24T21:03:09.803 DEBUG:teuthology.provision.fog.smithi055:Requesting image smithi_centos_9.stream (ID 10) 2024-03-24T21:03:09.846 DEBUG:teuthology.provision.fog.smithi194:Requesting image smithi_centos_9.stream (ID 10) 2024-03-24T21:03:09.860 INFO:teuthology.provision.fog.smithi055:Scheduling deploy of centos 9.stream 2024-03-24T21:03:09.892 INFO:teuthology.provision.fog.smithi194:Scheduling deploy of centos 9.stream 2024-03-24T21:03:10.122 INFO:teuthology.orchestra.console.smithi055:Power off 2024-03-24T21:03:10.122 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T21:03:10.135 INFO:teuthology.orchestra.console.smithi194:Power off 2024-03-24T21:03:10.136 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T21:03:10.148 DEBUG:teuthology.orchestra.console.smithi055:power off output: Chassis Power Control: Down/Off 2024-03-24T21:03:10.149 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T21:03:10.159 DEBUG:teuthology.orchestra.console.smithi194:power off output: Chassis Power Control: Down/Off 2024-03-24T21:03:10.159 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T21:03:10.179 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is off 2024-03-24T21:03:10.180 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2024-03-24T21:03:10.281 INFO:teuthology.orchestra.console.smithi194:Power off completed 2024-03-24T21:03:10.381 INFO:teuthology.orchestra.console.smithi194:Power on 2024-03-24T21:03:10.381 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-24T21:03:10.403 DEBUG:teuthology.orchestra.console.smithi194:power on output: Chassis Power Control: Up/On 2024-03-24T21:03:10.404 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T21:03:10.426 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is off 2024-03-24T21:03:14.182 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T21:03:14.292 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2024-03-24T21:03:14.393 INFO:teuthology.orchestra.console.smithi055:Power off completed 2024-03-24T21:03:14.427 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T21:03:14.494 INFO:teuthology.orchestra.console.smithi055:Power on 2024-03-24T21:03:14.494 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-24T21:03:14.514 DEBUG:teuthology.orchestra.console.smithi055:power on output: Chassis Power Control: Up/On 2024-03-24T21:03:14.515 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T21:03:14.536 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is off 2024-03-24T21:03:14.537 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2024-03-24T21:03:18.541 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T21:03:18.549 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T21:03:18.652 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-03-24T21:03:18.658 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2024-03-24T21:03:18.753 INFO:teuthology.orchestra.console.smithi194:Power on completed 2024-03-24T21:03:18.854 INFO:teuthology.provision.fog.smithi194:Waiting for deploy to finish 2024-03-24T21:03:22.662 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T21:03:22.770 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2024-03-24T21:03:26.775 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T21:03:26.883 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2024-03-24T21:03:30.886 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T21:03:30.995 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2024-03-24T21:03:34.997 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T21:03:35.107 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2024-03-24T21:03:39.110 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T21:03:39.219 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2024-03-24T21:03:43.224 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T21:03:43.332 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2024-03-24T21:03:47.334 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T21:03:47.444 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2024-03-24T21:03:51.446 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T21:03:51.554 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2024-03-24T21:03:55.559 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T21:03:55.668 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2024-03-24T21:03:59.672 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T21:03:59.780 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2024-03-24T21:04:03.784 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T21:04:03.894 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2024-03-24T21:04:07.898 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T21:04:08.007 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2024-03-24T21:04:12.012 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T21:04:12.121 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2024-03-24T21:04:12.122 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds 2024-03-24T21:04:12.222 DEBUG:teuthology.provision.fog.smithi055:Canceling deploy task with ID 890924 2024-03-24T21:04:12.384 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi055 2024-03-24T21:04:12.385 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T21:04:12.680 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=225dae8eb18047aca827b90b98b11b5e Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi055 2024-03-24T21:04:12.682 INFO:teuthology.nuke:Checking targets against current locks 2024-03-24T21:04:12.735 DEBUG:teuthology.nuke:shortname: smithi055 2024-03-24T21:04:12.735 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T21:04:12.758 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi055.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615310', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 21:03:08.537165', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIFDQ28C6530pPxDO2SWa8bs9/aC+YcNEK/30bhf9M6/yLvUOS5ZGNmIfFFNHPtbkBTm1kPdbtac6+qK4WAPrco='} 2024-03-24T21:04:12.761 DEBUG:teuthology.nuke:shortname: smithi194 2024-03-24T21:04:12.761 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T21:04:12.780 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-21_21:00:03-rados-squid-distro-default-smithi/7615310', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 21:03:08.535479', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNPC8YhkhfQHCOb5SGUMmL8cWQJ+1w0aJsoi2RHVZa86bw9zBMViE5fHtds6qjF5jt+yNNbMziBcjBq7MlHdEUs='} 2024-03-24T21:04:12.809 INFO:teuthology.orchestra.console.smithi055:Power off 2024-03-24T21:04:12.810 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T21:04:12.831 DEBUG:teuthology.orchestra.console.smithi055:power off output: Chassis Power Control: Down/Off 2024-03-24T21:04:12.832 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T21:04:12.847 INFO:teuthology.orchestra.console.smithi194:Power off 2024-03-24T21:04:12.847 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T21:04:12.854 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2024-03-24T21:04:12.868 DEBUG:teuthology.orchestra.console.smithi194:power off output: Chassis Power Control: Down/Off 2024-03-24T21:04:12.868 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T21:04:12.892 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-03-24T21:04:12.955 INFO:teuthology.orchestra.console.smithi055:Power off completed 2024-03-24T21:04:13.105 INFO:teuthology.lock.ops:unlocked: smithi055.front.sepia.ceph.com 2024-03-24T21:04:16.894 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T21:04:17.006 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is off 2024-03-24T21:04:17.107 INFO:teuthology.orchestra.console.smithi194:Power off completed 2024-03-24T21:04:17.255 INFO:teuthology.lock.ops:unlocked: smithi194.front.sepia.ceph.com 2024-03-24T21:04:17.503 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 58, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi055