2024-03-25T14:55:29.766 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T14:55:29.767 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T14:55:29.956 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe1145a5c10>, signals=[15]) 2024-03-25T14:55:29.956 INFO:teuthology.lock.ops:Start node 'smithi053.front.sepia.ceph.com' reimaging 2024-03-25T14:55:29.956 INFO:teuthology.lock.ops:Updating [smithi053.front.sepia.ceph.com]: reset os type and version on server 2024-03-25T14:55:29.956 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2024-03-25T14:55:29.976 INFO:teuthology.lock.ops:Start node 'smithi069.front.sepia.ceph.com' reimaging 2024-03-25T14:55:29.976 INFO:teuthology.lock.ops:Updating [smithi069.front.sepia.ceph.com]: reset os type and version on server 2024-03-25T14:55:29.976 INFO:teuthology.lock.ops:Updating smithi069.front.sepia.ceph.com on lock server 2024-03-25T14:55:30.098 DEBUG:teuthology.provision.fog.smithi053:Requesting image smithi_centos_9.stream (ID 10) 2024-03-25T14:55:30.103 DEBUG:teuthology.provision.fog.smithi069:Requesting image smithi_centos_9.stream (ID 10) 2024-03-25T14:55:30.151 INFO:teuthology.provision.fog.smithi053:Scheduling deploy of centos 9.stream 2024-03-25T14:55:30.154 INFO:teuthology.provision.fog.smithi069:Scheduling deploy of centos 9.stream 2024-03-25T14:55:30.410 INFO:teuthology.orchestra.console.smithi069:Power off 2024-03-25T14:55:30.411 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T14:55:30.429 INFO:teuthology.orchestra.console.smithi053:Power off 2024-03-25T14:55:30.430 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T14:55:30.438 DEBUG:teuthology.orchestra.console.smithi069:power off output: Chassis Power Control: Down/Off 2024-03-25T14:55:30.438 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:55:30.452 DEBUG:teuthology.orchestra.console.smithi053:power off output: Chassis Power Control: Down/Off 2024-03-25T14:55:30.452 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:55:30.477 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-03-25T14:55:30.477 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-03-25T14:55:34.482 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:55:34.489 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:55:34.592 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-25T14:55:34.598 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-03-25T14:55:34.693 INFO:teuthology.orchestra.console.smithi053:Power off completed 2024-03-25T14:55:34.794 INFO:teuthology.orchestra.console.smithi053:Power on 2024-03-25T14:55:34.794 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-25T14:55:34.816 DEBUG:teuthology.orchestra.console.smithi053:power on output: Chassis Power Control: Up/On 2024-03-25T14:55:34.816 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:55:34.840 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-25T14:55:38.602 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:55:38.711 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is off 2024-03-25T14:55:38.812 INFO:teuthology.orchestra.console.smithi069:Power off completed 2024-03-25T14:55:38.841 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:55:38.914 INFO:teuthology.orchestra.console.smithi069:Power on 2024-03-25T14:55:38.914 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-25T14:55:38.937 DEBUG:teuthology.orchestra.console.smithi069:power on output: Chassis Power Control: Up/On 2024-03-25T14:55:38.937 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:55:38.952 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-25T14:55:38.963 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is off 2024-03-25T14:55:42.955 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:55:42.964 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:55:43.065 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-25T14:55:43.075 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is off 2024-03-25T14:55:47.066 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:55:47.076 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:55:47.176 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-25T14:55:47.188 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-03-25T14:55:47.289 INFO:teuthology.orchestra.console.smithi069:Power on completed 2024-03-25T14:55:47.390 INFO:teuthology.provision.fog.smithi069:Waiting for deploy to finish 2024-03-25T14:55:51.179 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:55:51.288 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-25T14:55:55.292 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:55:55.404 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-25T14:55:59.406 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:55:59.517 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-25T14:56:03.518 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:56:03.627 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-25T14:56:07.631 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:56:07.741 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-25T14:56:11.742 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:56:11.851 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-25T14:56:15.854 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:56:15.964 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-25T14:56:19.966 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:56:20.075 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-25T14:56:24.078 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:56:24.188 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-25T14:56:28.192 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:56:28.302 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-25T14:56:32.306 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:56:32.416 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-25T14:56:32.416 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds 2024-03-25T14:56:32.517 DEBUG:teuthology.provision.fog.smithi053:Canceling deploy task with ID 894307 2024-03-25T14:56:32.570 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi053 2024-03-25T14:56:32.571 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T14:56:32.867 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f8631fc90c9b409a8be83bd705f7f4ad Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi053 2024-03-25T14:56:32.869 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T14:56:32.939 DEBUG:teuthology.nuke:shortname: smithi053 2024-03-25T14:56:32.939 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T14:56:32.960 DEBUG:teuthology.nuke:shortname: smithi069 2024-03-25T14:56:32.961 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T14:56:32.965 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/myoungwon-2024-03-15_14:24:24-rados-wip-ceph-dedup-daemon-distro-default-smithi/7603069', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 14:55:28.627421', 'locked_by': 'scheduled_myoungwon@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGqIo481x3wcFM0v8+64bWAww+JO0xwigE0sPR+C5bp87+Ct+ceOfyA/+1V7YCsxFytM+8SGVQhof1W3GrdLV50='} 2024-03-25T14:56:32.984 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/myoungwon-2024-03-15_14:24:24-rados-wip-ceph-dedup-daemon-distro-default-smithi/7603069', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 14:55:28.629428', 'locked_by': 'scheduled_myoungwon@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEFxKNv1X3qMcl6ZowXV3iuq68O+r5WXfaW8g+xjVKmbN6ttQRiOMt5TEVZ+4c8pWAWN7xYV92Fg+g5wGauxY64='} 2024-03-25T14:56:33.016 INFO:teuthology.orchestra.console.smithi053:Power off 2024-03-25T14:56:33.016 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T14:56:33.037 DEBUG:teuthology.orchestra.console.smithi053:power off output: Chassis Power Control: Down/Off 2024-03-25T14:56:33.038 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:56:33.057 INFO:teuthology.orchestra.console.smithi069:Power off 2024-03-25T14:56:33.057 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T14:56:33.070 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-25T14:56:33.083 DEBUG:teuthology.orchestra.console.smithi069:power off output: Chassis Power Control: Down/Off 2024-03-25T14:56:33.083 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:56:33.108 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-03-25T14:56:33.171 INFO:teuthology.orchestra.console.smithi053:Power off completed 2024-03-25T14:56:33.329 INFO:teuthology.lock.ops:unlocked: smithi053.front.sepia.ceph.com 2024-03-25T14:56:37.111 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T14:56:37.221 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is off 2024-03-25T14:56:37.322 INFO:teuthology.orchestra.console.smithi069:Power off completed 2024-03-25T14:56:37.479 INFO:teuthology.lock.ops:unlocked: smithi069.front.sepia.ceph.com 2024-03-25T14:56:37.732 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 58, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi053