2024-04-08T02:07:20.235 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-08T02:07:20.236 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T02:07:20.385 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff30f62ec10>, signals=[15]) 2024-04-08T02:07:20.385 INFO:teuthology.lock.ops:Start node 'smithi045.front.sepia.ceph.com' reimaging 2024-04-08T02:07:20.386 INFO:teuthology.lock.ops:Updating [smithi045.front.sepia.ceph.com]: reset os type and version on server 2024-04-08T02:07:20.386 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server 2024-04-08T02:07:20.410 INFO:teuthology.lock.ops:Start node 'smithi130.front.sepia.ceph.com' reimaging 2024-04-08T02:07:20.411 INFO:teuthology.lock.ops:Updating [smithi130.front.sepia.ceph.com]: reset os type and version on server 2024-04-08T02:07:20.411 INFO:teuthology.lock.ops:Updating smithi130.front.sepia.ceph.com on lock server 2024-04-08T02:07:20.525 DEBUG:teuthology.provision.fog.smithi045:Requesting image smithi_centos_9.stream (ID 10) 2024-04-08T02:07:20.551 DEBUG:teuthology.provision.fog.smithi130:Requesting image smithi_centos_9.stream (ID 10) 2024-04-08T02:07:20.574 INFO:teuthology.provision.fog.smithi045:Scheduling deploy of centos 9.stream 2024-04-08T02:07:20.599 INFO:teuthology.provision.fog.smithi130:Scheduling deploy of centos 9.stream 2024-04-08T02:07:20.859 INFO:teuthology.orchestra.console.smithi045:Power off 2024-04-08T02:07:20.860 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-08T02:07:20.872 INFO:teuthology.orchestra.console.smithi130:Power off 2024-04-08T02:07:20.872 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-08T02:07:20.884 DEBUG:teuthology.orchestra.console.smithi045:power off output: Chassis Power Control: Down/Off 2024-04-08T02:07:20.885 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T02:07:20.895 DEBUG:teuthology.orchestra.console.smithi130:power off output: Chassis Power Control: Down/Off 2024-04-08T02:07:20.896 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T02:07:20.913 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on 2024-04-08T02:07:20.917 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is on 2024-04-08T02:07:24.918 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T02:07:24.925 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T02:07:25.027 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on 2024-04-08T02:07:25.037 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is off 2024-04-08T02:07:25.138 INFO:teuthology.orchestra.console.smithi130:Power off completed 2024-04-08T02:07:25.238 INFO:teuthology.orchestra.console.smithi130:Power on 2024-04-08T02:07:25.239 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-04-08T02:07:25.262 DEBUG:teuthology.orchestra.console.smithi130:power on output: Chassis Power Control: Up/On 2024-04-08T02:07:25.263 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T02:07:25.286 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is off 2024-04-08T02:07:29.030 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T02:07:29.139 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off 2024-04-08T02:07:29.240 INFO:teuthology.orchestra.console.smithi045:Power off completed 2024-04-08T02:07:29.287 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T02:07:29.341 INFO:teuthology.orchestra.console.smithi045:Power on 2024-04-08T02:07:29.341 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-04-08T02:07:29.364 DEBUG:teuthology.orchestra.console.smithi045:power on output: Chassis Power Control: Up/On 2024-04-08T02:07:29.364 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T02:07:29.385 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off 2024-04-08T02:07:29.396 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is off 2024-04-08T02:07:33.390 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T02:07:33.397 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T02:07:33.500 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off 2024-04-08T02:07:33.507 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is off 2024-04-08T02:07:37.503 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T02:07:37.511 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T02:07:37.620 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is off 2024-04-08T02:07:37.734 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on 2024-04-08T02:07:37.834 INFO:teuthology.orchestra.console.smithi045:Power on completed 2024-04-08T02:07:37.935 INFO:teuthology.provision.fog.smithi045:Waiting for deploy to finish 2024-04-08T02:07:41.622 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T02:07:41.731 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is off 2024-04-08T02:07:45.736 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T02:07:45.845 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is off 2024-04-08T02:07:49.846 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T02:07:49.959 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is off 2024-04-08T02:07:53.960 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T02:07:54.069 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is off 2024-04-08T02:07:58.073 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T02:07:58.182 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is off 2024-04-08T02:08:02.186 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T02:08:02.296 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is off 2024-04-08T02:08:06.298 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T02:08:06.407 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is off 2024-04-08T02:08:10.411 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T02:08:10.521 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is off 2024-04-08T02:08:14.523 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T02:08:14.634 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is off 2024-04-08T02:08:18.638 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T02:08:18.747 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is off 2024-04-08T02:08:22.750 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T02:08:22.859 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is off 2024-04-08T02:08:22.860 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds 2024-04-08T02:08:22.960 DEBUG:teuthology.provision.fog.smithi130:Canceling deploy task with ID 934780 2024-04-08T02:08:23.051 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi130 2024-04-08T02:08:23.052 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T02:08:23.341 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c8e01f6fc0404b019f4a02fddc8f8023 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi130 2024-04-08T02:08:23.343 INFO:teuthology.nuke:Checking targets against current locks 2024-04-08T02:08:23.410 DEBUG:teuthology.nuke:shortname: smithi045 2024-04-08T02:08:23.410 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-08T02:08:23.433 DEBUG:teuthology.nuke:shortname: smithi130 2024-04-08T02:08:23.433 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-08T02:08:23.434 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-08_00:47:05-rados-wip-yuri-testing-2024-04-07-0902-distro-default-smithi/7645129', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-08 02:07:19.162080', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMIl200ouT9yixX54pWWReAhanhQTuP9W0iO7d4YJbRQX7nWWZ0RW3n8Lap4nXSPRX069Z47y36o/ZLmmvd27Xo='} 2024-04-08T02:08:23.460 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi130.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-08_00:47:05-rados-wip-yuri-testing-2024-04-07-0902-distro-default-smithi/7645129', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-08 02:07:19.160067', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN8u27mqnwCz0io+yKnzss/S7NZZkUQNpsXBdKP5C5U67Xa7M998iWknDSO+NIYkITey02of4Yh2zgE6F1Dczo4='} 2024-04-08T02:08:23.489 INFO:teuthology.orchestra.console.smithi045:Power off 2024-04-08T02:08:23.490 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-08T02:08:23.513 DEBUG:teuthology.orchestra.console.smithi045:power off output: Chassis Power Control: Down/Off 2024-04-08T02:08:23.514 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T02:08:23.531 INFO:teuthology.orchestra.console.smithi130:Power off 2024-04-08T02:08:23.531 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-08T02:08:23.540 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on 2024-04-08T02:08:23.552 DEBUG:teuthology.orchestra.console.smithi130:power off output: Chassis Power Control: Down/Off 2024-04-08T02:08:23.553 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T02:08:23.577 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is off 2024-04-08T02:08:23.678 INFO:teuthology.orchestra.console.smithi130:Power off completed 2024-04-08T02:08:23.835 INFO:teuthology.lock.ops:unlocked: smithi130.front.sepia.ceph.com 2024-04-08T02:08:27.542 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T02:08:27.651 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off 2024-04-08T02:08:27.752 INFO:teuthology.orchestra.console.smithi045:Power off completed 2024-04-08T02:08:27.909 INFO:teuthology.lock.ops:unlocked: smithi045.front.sepia.ceph.com 2024-04-08T02:08:28.168 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/dispatcher/supervisor.py", line 58, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi130