2023-07-27T01:44:28.754 INFO:root:teuthology version: 0.0.1.dev160+g407880c 2023-07-27T01:44:28.756 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-27T01:44:28.756 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-27T01:44:28.926 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0fc5331310>, signals=[15]) 2023-07-27T01:44:28.928 INFO:teuthology.lock.ops:Start node 'smithi089.front.sepia.ceph.com' reimaging 2023-07-27T01:44:28.928 INFO:teuthology.lock.ops:Updating [smithi089.front.sepia.ceph.com]: reset os type and version on server 2023-07-27T01:44:28.928 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2023-07-27T01:44:28.950 INFO:teuthology.lock.ops:Start node 'smithi172.front.sepia.ceph.com' reimaging 2023-07-27T01:44:28.951 INFO:teuthology.lock.ops:Updating [smithi172.front.sepia.ceph.com]: reset os type and version on server 2023-07-27T01:44:28.951 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2023-07-27T01:44:28.952 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /host {"name": "smithi089"} 2023-07-27T01:44:30.008 DEBUG:teuthology.provision.fog.smithi172:FOG request: GET /host {"name": "smithi172"} 2023-07-27T01:44:48.426 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /image {"name": "smithi_centos_8.stream"} 2023-07-27T01:44:58.459 DEBUG:teuthology.provision.fog.smithi172:FOG request: GET /image {"name": "smithi_centos_8.stream"} 2023-07-27T01:45:03.466 DEBUG:teuthology.provision.fog.smithi089:Requesting image smithi_centos_8.stream (ID 3) 2023-07-27T01:45:03.466 DEBUG:teuthology.provision.fog.smithi089:FOG request: PUT /host/194 {"imageID": 3} 2023-07-27T01:45:08.513 DEBUG:teuthology.provision.fog.smithi172:Requesting image smithi_centos_8.stream (ID 3) 2023-07-27T01:45:08.513 DEBUG:teuthology.provision.fog.smithi172:FOG request: PUT /host/277 {"imageID": 3} 2023-07-27T01:45:13.517 INFO:teuthology.provision.fog.smithi089:Scheduling deploy of centos 8.stream 2023-07-27T01:45:13.517 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None 2023-07-27T01:45:18.571 INFO:teuthology.provision.fog.smithi172:Scheduling deploy of centos 8.stream 2023-07-27T01:45:18.571 DEBUG:teuthology.provision.fog.smithi172:FOG request: GET /task/active None 2023-07-27T01:45:18.581 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /tasktype {"name": "deploy"} 2023-07-27T01:45:18.626 DEBUG:teuthology.provision.fog.smithi089:FOG request: POST /host/194/task {"taskTypeID": 1} 2023-07-27T01:45:18.636 DEBUG:teuthology.provision.fog.smithi172:FOG request: GET /tasktype {"name": "deploy"} 2023-07-27T01:45:18.667 DEBUG:teuthology.provision.fog.smithi172:FOG request: POST /host/277/task {"taskTypeID": 1} 2023-07-27T01:45:18.671 DEBUG:teuthology.provision.fog.smithi089:FOG request: GET /task/active None 2023-07-27T01:45:18.725 DEBUG:teuthology.provision.fog.smithi172:FOG request: GET /task/active None 2023-07-27T01:45:28.799 INFO:teuthology.orchestra.console:Power off smithi089 2023-07-27T01:45:28.800 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-27T01:45:28.815 INFO:teuthology.orchestra.console:Power off smithi172 2023-07-27T01:45:28.816 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-27T01:45:33.823 DEBUG:teuthology.orchestra.console:power off output: Chassis Power Control: Down/Off 2023-07-27T01:45:33.823 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:45:33.842 DEBUG:teuthology.orchestra.console:power off output: Chassis Power Control: Down/Off 2023-07-27T01:45:33.843 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:45:33.851 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on 2023-07-27T01:45:34.851 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:45:37.856 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:45:38.854 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:45:38.965 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:45:39.963 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:45:42.967 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:45:43.965 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:45:44.075 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off 2023-07-27T01:45:44.076 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:45:44.176 INFO:teuthology.orchestra.console:Power off for smithi172 completed 2023-07-27T01:45:44.277 INFO:teuthology.orchestra.console:Power on smithi172 2023-07-27T01:45:44.278 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-07-27T01:45:44.299 DEBUG:teuthology.orchestra.console:power on output: Chassis Power Control: Up/On 2023-07-27T01:45:44.299 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:45:44.321 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off 2023-07-27T01:45:48.080 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:45:48.323 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:45:48.434 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on 2023-07-27T01:45:48.535 INFO:teuthology.orchestra.console:Power on for smithi172 completed 2023-07-27T01:45:48.636 INFO:teuthology.provision.fog.smithi172:Waiting for deploy to finish 2023-07-27T01:45:48.637 DEBUG:teuthology.provision.fog.smithi172:FOG request: GET /task/active None 2023-07-27T01:45:48.714 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-27T01:45:49.192 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:45:53.193 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:45:54.304 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:45:58.308 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:45:59.418 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:46:03.420 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:46:04.529 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:46:07.088 WARNING:teuthology.provision.fog.smithi172:[Errno None] Unable to connect to port 22 on 172.21.15.172 2023-07-27T01:46:08.531 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:46:09.640 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:46:13.094 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-27T01:46:18.101 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:46:19.211 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:46:21.168 WARNING:teuthology.provision.fog.smithi172:[Errno None] Unable to connect to port 22 on 172.21.15.172 2023-07-27T01:46:23.212 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:46:24.320 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:46:27.172 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-27T01:46:32.177 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:46:33.287 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:46:35.248 WARNING:teuthology.provision.fog.smithi172:[Errno None] Unable to connect to port 22 on 172.21.15.172 2023-07-27T01:46:37.291 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:46:38.400 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:46:41.253 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-27T01:46:42.401 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:46:43.512 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:46:45.296 WARNING:teuthology.provision.fog.smithi172:[Errno None] Unable to connect to port 22 on 172.21.15.172 2023-07-27T01:46:47.513 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:46:47.625 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off 2023-07-27T01:46:47.726 INFO:teuthology.orchestra.console:Power off for smithi089 completed 2023-07-27T01:46:47.827 INFO:teuthology.orchestra.console:Power on smithi089 2023-07-27T01:46:47.827 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-07-27T01:46:51.299 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-27T01:46:52.852 DEBUG:teuthology.orchestra.console:power on output: Chassis Power Control: Up/On 2023-07-27T01:46:52.853 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:46:53.863 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:46:54.352 WARNING:teuthology.provision.fog.smithi172:[Errno None] Unable to connect to port 22 on 172.21.15.172 2023-07-27T01:46:57.868 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:46:58.978 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:47:00.355 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-27T01:47:02.982 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:47:03.408 WARNING:teuthology.provision.fog.smithi172:[Errno None] Unable to connect to port 22 on 172.21.15.172 2023-07-27T01:47:04.091 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:47:08.096 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:47:09.207 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:47:09.411 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-27T01:47:14.417 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:47:15.526 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:47:19.528 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:47:20.638 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:47:24.640 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:47:25.749 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:47:29.750 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:47:30.860 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:47:32.852 WARNING:teuthology.provision.fog.smithi172:[Errno None] Unable to connect to port 22 on 172.21.15.172 2023-07-27T01:47:34.863 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:47:35.972 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:47:38.857 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-27T01:47:38.972 ERROR:teuthology.orchestra.connection:Error authenticating with smithi172.front.sepia.ceph.com: Authentication failed. 2023-07-27T01:47:38.975 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-07-27T01:47:38.976 DEBUG:teuthology.orchestra.run.smithi172:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-07-27T01:47:38.977 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:47:39.974 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:47:39.984 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:47:40.991 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:47:41.085 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:47:41.997 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:47:43.003 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:47:44.008 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:47:45.014 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:47:45.086 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:47:46.021 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:47:46.197 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:47:47.027 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:47:48.032 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:47:49.037 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:47:50.041 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:47:50.224 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:47:51.047 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:47:51.334 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:47:52.053 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:47:53.058 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:47:54.064 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:47:55.071 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:47:55.335 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:47:56.077 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:47:56.446 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:47:57.133 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:47:58.139 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:47:59.144 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:48:00.150 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:48:00.531 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:48:01.156 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:48:01.642 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:48:02.163 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:48:03.169 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:48:04.175 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:48:05.180 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:48:05.644 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:48:06.186 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:48:06.754 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:48:06.755 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds 2023-07-27T01:48:06.755 DEBUG:teuthology.provision.fog.smithi089:Canceling deploy task with ID 413572 2023-07-27T01:48:06.756 DEBUG:teuthology.provision.fog.smithi089:FOG request: DELETE /task/cancel {"id": 413572} 2023-07-27T01:48:11.766 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:48:11.824 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi089 2023-07-27T01:48:11.825 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-27T01:48:11.825 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-27T01:48:12.147 INFO:teuthology.nuke:Checking targets against current locks 2023-07-27T01:48:12.192 DEBUG:teuthology.nuke:shortname: smithi089 2023-07-27T01:48:12.192 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-27T01:48:12.216 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-26_15:54:22-rados-wip-yuri6-testing-2023-07-24-0819-pacific-distro-default-smithi/7353392', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-27 01:44:27.920956', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOcf/Pw95vjx6AXC+0UkCvBxkuOgOtpbByymCMWWB56Syu0808Gju8EyE52mDJCSkwXnsHVFTgwWlu0v+7nyJhtF6tBhXeTgDe4DIC+9r4Ut33zZH/8oAODr3dmyUm+Cuh2tAf8fPFWMmRJ4A0B25IckiAZB090DaR/Uzg2x41Boq1HfksbF73K+kHsp228eeiBGXori4TEgehRFCVCfiB328sLUi3sePo86f1v0LM9w8oM78vJ9kYVd88N38tlx1O8wKW/+BHk+otUjgJAlSqAfVoDb54abVlGddvR1RezdvPf9Uq/uSnJCni8cKUtKHpNX2VH2f28AIoQiBHaV/fMUAG+wXY+C6Oe86NZTNEZNxP3mrRjxY5PCE0tdnjIbGV9QGg9tofGdSVEmd5Czkfqg67eP8xH01tI3qZIJnDqZDeLyzsD1MnDHiRHtGZmMrlF4/6puXUDv4tIbtyfX0ryS2Ubd2ceAF0PzpzODY/K/aiARNjgO9Dcvd+z7zwFFc='} 2023-07-27T01:48:12.220 DEBUG:teuthology.nuke:shortname: smithi172 2023-07-27T01:48:12.221 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-27T01:48:12.247 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-26_15:54:22-rados-wip-yuri6-testing-2023-07-24-0819-pacific-distro-default-smithi/7353392', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-27 01:44:27.919264', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0='} 2023-07-27T01:48:12.273 INFO:teuthology.orchestra.console:Power off smithi089 2023-07-27T01:48:12.274 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-27T01:48:12.312 INFO:teuthology.orchestra.console:Power off smithi172 2023-07-27T01:48:12.313 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-27T01:48:12.336 DEBUG:teuthology.orchestra.console:power off output: Chassis Power Control: Down/Off 2023-07-27T01:48:12.338 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:48:12.359 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on 2023-07-27T01:48:12.802 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:48:13.807 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:48:14.812 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:48:15.817 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:48:16.017 DEBUG:teuthology.orchestra.console:power off output: Chassis Power Control: Down/Off 2023-07-27T01:48:16.017 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:48:16.057 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on 2023-07-27T01:48:16.361 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:48:16.471 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on 2023-07-27T01:48:16.831 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:48:17.836 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:48:18.841 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:48:19.847 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-07-27T01:48:20.059 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:48:20.473 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:48:21.169 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:48:21.583 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:48:25.170 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:48:25.584 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:48:26.280 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:48:26.697 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:48:30.282 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:48:30.698 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:48:31.392 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:48:31.808 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:48:35.393 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:48:35.809 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:48:36.504 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:48:36.922 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:48:40.506 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:48:40.923 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:48:41.617 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:48:42.033 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:48:45.619 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:48:45.729 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off 2023-07-27T01:48:45.830 INFO:teuthology.orchestra.console:Power off for smithi089 completed 2023-07-27T01:48:50.936 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:48:54.441 INFO:teuthology.lock.ops:unlocked: smithi089.front.sepia.ceph.com 2023-07-27T01:48:55.407 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:48:59.408 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:49:00.520 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:49:04.521 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:49:05.633 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:49:09.635 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:49:10.746 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:49:14.748 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:49:15.862 DEBUG:teuthology.orchestra.console:check power output: 2023-07-27T01:49:19.863 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-27T01:49:19.974 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off 2023-07-27T01:49:20.075 INFO:teuthology.orchestra.console:Power off for smithi172 completed 2023-07-27T01:49:30.243 INFO:teuthology.lock.ops:unlocked: smithi172.front.sepia.ceph.com 2023-07-27T01:49:31.400 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/dispatcher/supervisor.py", line 57, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/provision/fog.py", line 85, in create self.remote.console.power_on() File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/orchestra/console.py", line 252, in power_on raise RuntimeError(err_msg) RuntimeError: Failed to power on smithi089