2024-03-27T11:06:10.317 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-27T11:06:10.318 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-27T11:06:10.452 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe52f304c10>, signals=[15]) 2024-03-27T11:06:10.452 INFO:teuthology.lock.ops:Start node 'smithi022.front.sepia.ceph.com' reimaging 2024-03-27T11:06:10.453 INFO:teuthology.lock.ops:Updating [smithi022.front.sepia.ceph.com]: reset os type and version on server 2024-03-27T11:06:10.453 INFO:teuthology.lock.ops:Updating smithi022.front.sepia.ceph.com on lock server 2024-03-27T11:06:10.470 INFO:teuthology.lock.ops:Start node 'smithi145.front.sepia.ceph.com' reimaging 2024-03-27T11:06:10.470 INFO:teuthology.lock.ops:Updating [smithi145.front.sepia.ceph.com]: reset os type and version on server 2024-03-27T11:06:10.471 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server 2024-03-27T11:06:10.588 DEBUG:teuthology.provision.fog.smithi022:Requesting image smithi_ubuntu_20.04 (ID 2) 2024-03-27T11:06:10.602 DEBUG:teuthology.provision.fog.smithi145:Requesting image smithi_ubuntu_20.04 (ID 2) 2024-03-27T11:06:10.631 INFO:teuthology.provision.fog.smithi022:Scheduling deploy of ubuntu 20.04 2024-03-27T11:06:10.650 INFO:teuthology.provision.fog.smithi145:Scheduling deploy of ubuntu 20.04 2024-03-27T11:06:10.875 INFO:teuthology.orchestra.console.smithi022:Power off 2024-03-27T11:06:10.876 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-27T11:06:10.898 DEBUG:teuthology.orchestra.console.smithi022:power off output: Chassis Power Control: Down/Off 2024-03-27T11:06:10.898 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T11:06:10.913 INFO:teuthology.orchestra.console.smithi145:Power off 2024-03-27T11:06:10.913 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-27T11:06:10.923 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is on 2024-03-27T11:06:10.934 DEBUG:teuthology.orchestra.console.smithi145:power off output: Chassis Power Control: Down/Off 2024-03-27T11:06:10.934 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T11:06:10.955 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is off 2024-03-27T11:06:11.056 INFO:teuthology.orchestra.console.smithi145:Power off completed 2024-03-27T11:06:11.156 INFO:teuthology.orchestra.console.smithi145:Power on 2024-03-27T11:06:11.157 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-27T11:06:11.181 DEBUG:teuthology.orchestra.console.smithi145:power on output: Chassis Power Control: Up/On 2024-03-27T11:06:11.181 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T11:06:11.206 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is off 2024-03-27T11:06:14.928 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T11:06:15.036 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is off 2024-03-27T11:06:15.137 INFO:teuthology.orchestra.console.smithi022:Power off completed 2024-03-27T11:06:15.206 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T11:06:15.237 INFO:teuthology.orchestra.console.smithi022:Power on 2024-03-27T11:06:15.238 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-27T11:06:15.258 DEBUG:teuthology.orchestra.console.smithi022:power on output: Chassis Power Control: Up/On 2024-03-27T11:06:15.258 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T11:06:15.279 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is off 2024-03-27T11:06:15.315 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is off 2024-03-27T11:06:19.282 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T11:06:19.316 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T11:06:19.391 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is off 2024-03-27T11:06:19.425 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is on 2024-03-27T11:06:19.525 INFO:teuthology.orchestra.console.smithi145:Power on completed 2024-03-27T11:06:19.626 INFO:teuthology.provision.fog.smithi145:Waiting for deploy to finish 2024-03-27T11:06:23.393 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T11:06:23.502 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is off 2024-03-27T11:06:27.506 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T11:06:27.615 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is off 2024-03-27T11:06:31.621 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T11:06:31.730 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is off 2024-03-27T11:06:35.731 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T11:06:35.840 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is off 2024-03-27T11:06:39.843 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T11:06:39.953 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is off 2024-03-27T11:06:43.957 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T11:06:44.066 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is off 2024-03-27T11:06:48.070 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T11:06:48.179 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is off 2024-03-27T11:06:52.182 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T11:06:52.291 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is off 2024-03-27T11:06:56.294 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T11:06:56.403 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is off 2024-03-27T11:07:00.406 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T11:07:00.515 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is off 2024-03-27T11:07:04.520 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T11:07:04.628 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is off 2024-03-27T11:07:08.630 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T11:07:08.738 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is off 2024-03-27T11:07:12.742 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T11:07:12.851 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is off 2024-03-27T11:07:12.852 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds 2024-03-27T11:07:12.952 DEBUG:teuthology.provision.fog.smithi022:Canceling deploy task with ID 900442 2024-03-27T11:07:13.167 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi022 2024-03-27T11:07:13.169 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-27T11:07:13.461 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f34778999d40408da3b9c5194324bbd3 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi022 2024-03-27T11:07:13.463 INFO:teuthology.nuke:Checking targets against current locks 2024-03-27T11:07:13.518 DEBUG:teuthology.nuke:shortname: smithi022 2024-03-27T11:07:13.518 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-27T11:07:13.543 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi022.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/svelar-2024-03-27_06:35:39-rados-wip-svelar-testing-reef-2024-03-26-0716-distro-default-smithi/7625835', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-27 11:06:09.291637', 'locked_by': 'scheduled_svelar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHcPOaqVNUE1AMbo8X4/JmNBiydOfhoMbbyZD3sAY8EKsX6degvILGT1a4g6rqDM0cMlFlvMHyKx7GkzWQnTwrc='} 2024-03-27T11:07:13.546 DEBUG:teuthology.nuke:shortname: smithi145 2024-03-27T11:07:13.546 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-27T11:07:13.567 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi145.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/svelar-2024-03-27_06:35:39-rados-wip-svelar-testing-reef-2024-03-26-0716-distro-default-smithi/7625835', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-27 11:06:09.289664', 'locked_by': 'scheduled_svelar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH2rx34Ykad/6TSWZPnsHqrhvn2IXlVtolfL9pPs4vqtYyhU0SRlh8AVH9G4K4+QF+6gU909MFsGHEv4GJXOE4s='} 2024-03-27T11:07:13.588 INFO:teuthology.orchestra.console.smithi022:Power off 2024-03-27T11:07:13.589 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-27T11:07:13.609 DEBUG:teuthology.orchestra.console.smithi022:power off output: Chassis Power Control: Down/Off 2024-03-27T11:07:13.609 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T11:07:13.626 INFO:teuthology.orchestra.console.smithi145:Power off 2024-03-27T11:07:13.627 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-27T11:07:13.635 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is off 2024-03-27T11:07:13.647 DEBUG:teuthology.orchestra.console.smithi145:power off output: Chassis Power Control: Down/Off 2024-03-27T11:07:13.647 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T11:07:13.668 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is on 2024-03-27T11:07:13.736 INFO:teuthology.orchestra.console.smithi022:Power off completed 2024-03-27T11:07:13.887 INFO:teuthology.lock.ops:unlocked: smithi022.front.sepia.ceph.com 2024-03-27T11:07:17.671 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-27T11:07:17.780 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is off 2024-03-27T11:07:17.881 INFO:teuthology.orchestra.console.smithi145:Power off completed 2024-03-27T11:07:18.037 INFO:teuthology.lock.ops:unlocked: smithi145.front.sepia.ceph.com 2024-03-27T11:07:18.299 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 58, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi022