2023-12-01T05:38:37.772 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-01T05:38:37.772 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-01T05:38:37.909 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9239db9550>, signals=[15]) 2023-12-01T05:38:37.910 INFO:teuthology.lock.ops:Start node 'smithi005.front.sepia.ceph.com' reimaging 2023-12-01T05:38:37.910 INFO:teuthology.lock.ops:Updating [smithi005.front.sepia.ceph.com]: reset os type and version on server 2023-12-01T05:38:37.910 INFO:teuthology.lock.ops:Updating smithi005.front.sepia.ceph.com on lock server 2023-12-01T05:38:37.932 INFO:teuthology.lock.ops:Start node 'smithi016.front.sepia.ceph.com' reimaging 2023-12-01T05:38:37.932 INFO:teuthology.lock.ops:Updating [smithi016.front.sepia.ceph.com]: reset os type and version on server 2023-12-01T05:38:37.933 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2023-12-01T05:38:38.054 DEBUG:teuthology.provision.fog.smithi005:Requesting image smithi_centos_8.stream (ID 3) 2023-12-01T05:38:38.076 DEBUG:teuthology.provision.fog.smithi016:Requesting image smithi_centos_8.stream (ID 3) 2023-12-01T05:38:38.107 INFO:teuthology.provision.fog.smithi005:Scheduling deploy of centos 8.stream 2023-12-01T05:38:38.128 INFO:teuthology.provision.fog.smithi016:Scheduling deploy of centos 8.stream 2023-12-01T05:38:38.399 INFO:teuthology.orchestra.console.smithi005:Power off 2023-12-01T05:38:38.444 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-01T05:38:38.458 INFO:teuthology.orchestra.console.smithi016:Power off 2023-12-01T05:38:38.459 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-01T05:38:38.475 DEBUG:teuthology.orchestra.console.smithi005:power off output: Chassis Power Control: Down/Off 2023-12-01T05:38:38.476 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T05:38:38.490 DEBUG:teuthology.orchestra.console.smithi016:power off output: Chassis Power Control: Down/Off 2023-12-01T05:38:38.491 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T05:38:38.501 DEBUG:teuthology.orchestra.console.smithi005:check power output: Chassis Power is off 2023-12-01T05:38:38.513 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is off 2023-12-01T05:38:38.602 INFO:teuthology.orchestra.console.smithi005:Power off completed 2023-12-01T05:38:38.614 INFO:teuthology.orchestra.console.smithi016:Power off completed 2023-12-01T05:38:38.703 INFO:teuthology.orchestra.console.smithi005:Power on 2023-12-01T05:38:38.703 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-12-01T05:38:38.715 INFO:teuthology.orchestra.console.smithi016:Power on 2023-12-01T05:38:38.715 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-12-01T05:38:38.730 DEBUG:teuthology.orchestra.console.smithi005:power on output: Chassis Power Control: Up/On 2023-12-01T05:38:38.730 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T05:38:38.742 DEBUG:teuthology.orchestra.console.smithi016:power on output: Chassis Power Control: Up/On 2023-12-01T05:38:38.742 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T05:38:38.755 DEBUG:teuthology.orchestra.console.smithi005:check power output: Chassis Power is off 2023-12-01T05:38:38.765 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is off 2023-12-01T05:38:42.758 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T05:38:42.766 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T05:38:42.875 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is off 2023-12-01T05:38:43.866 DEBUG:teuthology.orchestra.console.smithi005:check power output: 2023-12-01T05:38:46.878 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T05:38:46.989 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is on 2023-12-01T05:38:47.090 INFO:teuthology.orchestra.console.smithi016:Power on completed 2023-12-01T05:38:47.190 INFO:teuthology.provision.fog.smithi016:Waiting for deploy to finish 2023-12-01T05:38:47.868 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T05:38:48.977 DEBUG:teuthology.orchestra.console.smithi005:check power output: 2023-12-01T05:38:52.978 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T05:38:54.089 DEBUG:teuthology.orchestra.console.smithi005:check power output: 2023-12-01T05:38:58.092 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T05:38:59.203 DEBUG:teuthology.orchestra.console.smithi005:check power output: 2023-12-01T05:39:03.205 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T05:39:04.314 DEBUG:teuthology.orchestra.console.smithi005:check power output: 2023-12-01T05:39:08.318 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T05:39:09.427 DEBUG:teuthology.orchestra.console.smithi005:check power output: 2023-12-01T05:39:13.430 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T05:39:14.540 DEBUG:teuthology.orchestra.console.smithi005:check power output: 2023-12-01T05:39:18.541 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T05:39:19.653 DEBUG:teuthology.orchestra.console.smithi005:check power output: 2023-12-01T05:39:23.654 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T05:39:24.762 DEBUG:teuthology.orchestra.console.smithi005:check power output: 2023-12-01T05:39:28.766 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T05:39:29.876 DEBUG:teuthology.orchestra.console.smithi005:check power output: 2023-12-01T05:39:33.878 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T05:39:33.987 DEBUG:teuthology.orchestra.console.smithi005:check power output: Chassis Power is off 2023-12-01T05:39:37.990 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T05:39:38.099 DEBUG:teuthology.orchestra.console.smithi005:check power output: Chassis Power is off 2023-12-01T05:39:42.103 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T05:39:42.213 DEBUG:teuthology.orchestra.console.smithi005:check power output: Chassis Power is off 2023-12-01T05:39:46.218 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T05:39:46.329 DEBUG:teuthology.orchestra.console.smithi005:check power output: Chassis Power is off 2023-12-01T05:39:46.330 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds 2023-12-01T05:39:46.430 DEBUG:teuthology.provision.fog.smithi005:Canceling deploy task with ID 634588 2023-12-01T05:39:46.544 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi005 2023-12-01T05:39:46.546 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-01T05:39:46.706 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=671543442be8443ba08cca9e4a20f0e0 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi005 2023-12-01T05:39:46.708 INFO:teuthology.nuke:Checking targets against current locks 2023-12-01T05:39:46.770 DEBUG:teuthology.nuke:shortname: smithi005 2023-12-01T05:39:46.770 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-01T05:39:46.791 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi005.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/phlogistonjohn-2023-11-30_21:17:15-orch:cephadm-wip-phlogistonjohn-testing-2023-11-30-1010-distro-default-smithi/7472972', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-01 05:38:36.957775', 'locked_by': 'scheduled_phlogistonjohn@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-12-01T05:39:46.795 DEBUG:teuthology.nuke:shortname: smithi016 2023-12-01T05:39:46.796 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-01T05:39:46.818 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/phlogistonjohn-2023-11-30_21:17:15-orch:cephadm-wip-phlogistonjohn-testing-2023-11-30-1010-distro-default-smithi/7472972', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-01 05:38:36.956622', 'locked_by': 'scheduled_phlogistonjohn@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-12-01T05:39:46.846 INFO:teuthology.orchestra.console.smithi005:Power off 2023-12-01T05:39:46.847 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-01T05:39:46.874 DEBUG:teuthology.orchestra.console.smithi005:power off output: Chassis Power Control: Down/Off 2023-12-01T05:39:46.874 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T05:39:46.900 INFO:teuthology.orchestra.console.smithi016:Power off 2023-12-01T05:39:46.900 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-01T05:39:46.917 DEBUG:teuthology.orchestra.console.smithi005:check power output: Chassis Power is off 2023-12-01T05:39:46.932 DEBUG:teuthology.orchestra.console.smithi016:power off output: Chassis Power Control: Down/Off 2023-12-01T05:39:46.932 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T05:39:46.967 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is on 2023-12-01T05:39:47.018 INFO:teuthology.orchestra.console.smithi005:Power off completed 2023-12-01T05:39:47.180 INFO:teuthology.lock.ops:unlocked: smithi005.front.sepia.ceph.com 2023-12-01T05:39:50.970 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T05:39:51.083 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is off 2023-12-01T05:39:51.184 INFO:teuthology.orchestra.console.smithi016:Power off completed 2023-12-01T05:39:51.335 INFO:teuthology.lock.ops:unlocked: smithi016.front.sepia.ceph.com 2023-12-01T05:39:51.593 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/dispatcher/supervisor.py", line 58, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi005