2023-07-25T13:58:30.419 INFO:root:teuthology version: 0.0.1.dev160+g407880c
2023-07-25T13:58:30.420 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-25T13:58:30.421 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-25T13:58:45.372 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f9e4ed62d90>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f9e4e7235e0>, signals=[15])
2023-07-25T13:58:45.374 INFO:teuthology.lock.ops:Start node 'smithi002.front.sepia.ceph.com' reimaging
2023-07-25T13:58:45.374 INFO:teuthology.lock.ops:Updating [smithi002.front.sepia.ceph.com]: reset os type and version on server
2023-07-25T13:58:45.374 INFO:teuthology.lock.ops:Updating smithi002.front.sepia.ceph.com on lock server
2023-07-25T13:58:50.402 INFO:teuthology.lock.ops:Start node 'smithi087.front.sepia.ceph.com' reimaging
2023-07-25T13:58:50.402 INFO:teuthology.lock.ops:Updating [smithi087.front.sepia.ceph.com]: reset os type and version on server
2023-07-25T13:58:50.402 INFO:teuthology.lock.ops:Updating smithi087.front.sepia.ceph.com on lock server
2023-07-25T13:58:50.404 DEBUG:teuthology.provision.fog.smithi002:FOG request: GET /host {"name": "smithi002"}
2023-07-25T13:58:54.385 INFO:teuthology.lock.ops:Start node 'smithi140.front.sepia.ceph.com' reimaging
2023-07-25T13:58:54.385 INFO:teuthology.lock.ops:Updating [smithi140.front.sepia.ceph.com]: reset os type and version on server
2023-07-25T13:58:54.385 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server
2023-07-25T13:58:54.387 DEBUG:teuthology.provision.fog.smithi087:FOG request: GET /host {"name": "smithi087"}
2023-07-25T13:59:10.450 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /host {"name": "smithi140"}
2023-07-25T13:59:20.466 DEBUG:teuthology.provision.fog.smithi002:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-07-25T13:59:30.483 DEBUG:teuthology.provision.fog.smithi087:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-07-25T13:59:30.512 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-07-25T13:59:30.525 DEBUG:teuthology.provision.fog.smithi002:Requesting image smithi_rhel_8.6 (ID 5)
2023-07-25T13:59:30.525 DEBUG:teuthology.provision.fog.smithi002:FOG request: PUT /host/108 {"imageID": 5}
2023-07-25T13:59:30.528 DEBUG:teuthology.provision.fog.smithi087:Requesting image smithi_rhel_8.6 (ID 5)
2023-07-25T13:59:30.529 DEBUG:teuthology.provision.fog.smithi087:FOG request: PUT /host/192 {"imageID": 5}
2023-07-25T13:59:30.553 DEBUG:teuthology.provision.fog.smithi140:Requesting image smithi_rhel_8.6 (ID 5)
2023-07-25T13:59:30.553 DEBUG:teuthology.provision.fog.smithi140:FOG request: PUT /host/245 {"imageID": 5}
2023-07-25T13:59:30.569 INFO:teuthology.provision.fog.smithi002:Scheduling deploy of rhel 8.6
2023-07-25T13:59:30.570 DEBUG:teuthology.provision.fog.smithi002:FOG request: GET /task/active None
2023-07-25T13:59:30.573 INFO:teuthology.provision.fog.smithi087:Scheduling deploy of rhel 8.6
2023-07-25T13:59:30.573 DEBUG:teuthology.provision.fog.smithi087:FOG request: GET /task/active None
2023-07-25T13:59:30.597 INFO:teuthology.provision.fog.smithi140:Scheduling deploy of rhel 8.6
2023-07-25T13:59:30.598 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-07-25T13:59:30.634 DEBUG:teuthology.provision.fog.smithi002:FOG request: GET /tasktype {"name": "deploy"}
2023-07-25T13:59:30.641 DEBUG:teuthology.provision.fog.smithi087:FOG request: GET /tasktype {"name": "deploy"}
2023-07-25T13:59:30.661 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /tasktype {"name": "deploy"}
2023-07-25T13:59:30.668 DEBUG:teuthology.provision.fog.smithi002:FOG request: POST /host/108/task {"taskTypeID": 1}
2023-07-25T13:59:30.678 DEBUG:teuthology.provision.fog.smithi087:FOG request: POST /host/192/task {"taskTypeID": 1}
2023-07-25T13:59:30.693 DEBUG:teuthology.provision.fog.smithi140:FOG request: POST /host/245/task {"taskTypeID": 1}
2023-07-25T13:59:30.730 DEBUG:teuthology.provision.fog.smithi002:FOG request: GET /task/active None
2023-07-25T13:59:30.739 DEBUG:teuthology.provision.fog.smithi087:FOG request: GET /task/active None
2023-07-25T13:59:30.745 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-07-25T13:59:41.528 INFO:teuthology.orchestra.console:Power off smithi087
2023-07-25T13:59:41.529 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-25T13:59:41.554 INFO:teuthology.orchestra.console:Power off smithi140
2023-07-25T13:59:41.555 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-25T13:59:41.572 INFO:teuthology.orchestra.console:Power off smithi002
2023-07-25T13:59:41.573 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-25T13:59:41.586 DEBUG:teuthology.orchestra.console:power off output: Chassis Power Control: Down/Off
2023-07-25T13:59:41.587 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T13:59:41.597 DEBUG:teuthology.orchestra.console:power off output: Chassis Power Control: Down/Off
2023-07-25T13:59:41.598 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T13:59:41.610 DEBUG:teuthology.orchestra.console:power off output: Chassis Power Control: Down/Off
2023-07-25T13:59:41.611 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T13:59:41.622 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off
2023-07-25T13:59:41.623 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-25T13:59:41.637 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-25T13:59:41.724 INFO:teuthology.orchestra.console:Power off for smithi087 completed
2023-07-25T13:59:41.824 INFO:teuthology.orchestra.console:Power on smithi087
2023-07-25T13:59:41.825 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-25T13:59:41.846 DEBUG:teuthology.orchestra.console:power on output: Chassis Power Control: Up/On
2023-07-25T13:59:41.847 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T13:59:41.871 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off
2023-07-25T13:59:45.628 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T13:59:45.639 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T13:59:45.739 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-25T13:59:45.750 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-25T13:59:45.872 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T13:59:46.982 DEBUG:teuthology.orchestra.console:check power output: 
2023-07-25T13:59:49.744 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T13:59:49.753 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T13:59:49.856 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-25T13:59:49.863 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off
2023-07-25T13:59:49.963 INFO:teuthology.orchestra.console:Power off for smithi002 completed
2023-07-25T13:59:50.064 INFO:teuthology.orchestra.console:Power on smithi002
2023-07-25T13:59:50.065 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-25T13:59:50.086 DEBUG:teuthology.orchestra.console:power on output: Chassis Power Control: Up/On
2023-07-25T13:59:50.087 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T13:59:50.111 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off
2023-07-25T13:59:50.984 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T13:59:52.096 DEBUG:teuthology.orchestra.console:check power output: 
2023-07-25T13:59:53.859 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T13:59:54.112 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T13:59:54.972 DEBUG:teuthology.orchestra.console:check power output: 
2023-07-25T13:59:55.221 DEBUG:teuthology.orchestra.console:check power output: 
2023-07-25T13:59:56.098 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T13:59:56.208 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-25T13:59:56.309 INFO:teuthology.orchestra.console:Power on for smithi087 completed
2023-07-25T13:59:56.410 INFO:teuthology.provision.fog.smithi087:Waiting for deploy to finish
2023-07-25T13:59:56.411 DEBUG:teuthology.provision.fog.smithi087:FOG request: GET /task/active None
2023-07-25T14:00:01.416 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:00:01.425 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:00:01.490 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-25T14:00:06.496 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off
2023-07-25T14:00:06.496 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-25T14:00:06.597 INFO:teuthology.orchestra.console:Power off for smithi140 completed
2023-07-25T14:00:06.598 INFO:teuthology.orchestra.console:Power on for smithi002 completed
2023-07-25T14:00:06.698 INFO:teuthology.orchestra.console:Power on smithi140
2023-07-25T14:00:06.699 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-25T14:00:06.710 INFO:teuthology.provision.fog.smithi002:Waiting for deploy to finish
2023-07-25T14:00:06.711 DEBUG:teuthology.provision.fog.smithi002:FOG request: GET /task/active None
2023-07-25T14:00:10.898 WARNING:teuthology.provision.fog.smithi087:[Errno None] Unable to connect to port 22 on 172.21.15.87
2023-07-25T14:00:16.731 DEBUG:teuthology.orchestra.console:power on output: Chassis Power Control: Up/On
2023-07-25T14:00:16.731 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:00:16.901 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-25T14:00:17.740 DEBUG:teuthology.orchestra.console:check power output: 
2023-07-25T14:00:19.984 WARNING:teuthology.provision.fog.smithi087:[Errno None] Unable to connect to port 22 on 172.21.15.87
2023-07-25T14:00:21.743 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:00:22.855 DEBUG:teuthology.orchestra.console:check power output: 
2023-07-25T14:00:25.956 DEBUG:teuthology.provision.fog.smithi002:FOG request: GET /task/active None
2023-07-25T14:00:30.965 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-25T14:00:35.969 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:00:37.080 DEBUG:teuthology.orchestra.console:check power output: 
2023-07-25T14:00:39.024 WARNING:teuthology.provision.fog.smithi087:[Errno None] Unable to connect to port 22 on 172.21.15.87
2023-07-25T14:00:41.082 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:00:42.192 DEBUG:teuthology.orchestra.console:check power output: 
2023-07-25T14:00:45.029 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-25T14:00:50.035 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:00:51.035 DEBUG:teuthology.provision.fog.smithi002:FOG request: GET /task/active None
2023-07-25T14:00:56.044 WARNING:teuthology.provision.fog.smithi087:[Errno None] Unable to connect to port 22 on 172.21.15.87
2023-07-25T14:00:56.044 DEBUG:teuthology.orchestra.console:check power output: 
2023-07-25T14:01:00.048 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:01:01.160 DEBUG:teuthology.orchestra.console:check power output: 
2023-07-25T14:01:02.047 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-25T14:01:07.053 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:01:08.164 DEBUG:teuthology.orchestra.console:check power output: 
2023-07-25T14:01:10.128 WARNING:teuthology.provision.fog.smithi087:[Errno None] Unable to connect to port 22 on 172.21.15.87
2023-07-25T14:01:11.099 DEBUG:teuthology.provision.fog.smithi002:FOG request: GET /task/active None
2023-07-25T14:01:16.105 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:01:16.130 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-25T14:01:17.216 DEBUG:teuthology.orchestra.console:check power output: 
2023-07-25T14:01:19.184 WARNING:teuthology.provision.fog.smithi087:[Errno None] Unable to connect to port 22 on 172.21.15.87
2023-07-25T14:01:21.219 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:01:22.328 DEBUG:teuthology.orchestra.console:check power output: 
2023-07-25T14:01:25.189 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-25T14:01:26.144 WARNING:teuthology.provision.fog.smithi087:[Errno None] Unable to connect to port 22 on 172.21.15.87
2023-07-25T14:01:26.331 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:01:27.440 DEBUG:teuthology.orchestra.console:check power output: 
2023-07-25T14:01:31.173 DEBUG:teuthology.provision.fog.smithi002:FOG request: GET /task/active None
2023-07-25T14:01:34.141 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:01:34.149 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-25T14:01:34.265 ERROR:teuthology.orchestra.connection:Error authenticating with smithi087.front.sepia.ceph.com: Authentication failed.
2023-07-25T14:01:34.267 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-07-25T14:01:34.268 DEBUG:teuthology.orchestra.run.smithi087:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-07-25T14:01:34.269 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:01:35.251 DEBUG:teuthology.orchestra.console:check power output: 
2023-07-25T14:01:35.275 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:01:36.281 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:01:37.287 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:01:38.293 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:01:39.251 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:01:39.300 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:01:40.307 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:01:40.363 DEBUG:teuthology.orchestra.console:check power output: 
2023-07-25T14:01:41.312 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:01:42.320 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:01:43.325 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:01:44.331 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:01:44.364 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:01:45.338 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:01:45.475 DEBUG:teuthology.orchestra.console:check power output: 
2023-07-25T14:01:46.344 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:01:47.350 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:01:48.355 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:01:49.219 DEBUG:teuthology.provision.fog.smithi002:FOG request: GET /task/active None
2023-07-25T14:01:50.403 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:01:50.414 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:01:51.419 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:01:51.515 DEBUG:teuthology.orchestra.console:check power output: 
2023-07-25T14:01:52.426 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:01:53.431 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:01:54.439 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:01:55.516 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:01:55.536 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:01:56.543 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:01:56.627 DEBUG:teuthology.orchestra.console:check power output: 
2023-07-25T14:01:56.628 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-07-25T14:01:56.629 DEBUG:teuthology.provision.fog.smithi140:Canceling deploy task with ID None
2023-07-25T14:01:56.629 DEBUG:teuthology.provision.fog.smithi140:FOG request: DELETE /task/cancel {"id": None}
2023-07-25T14:02:01.638 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:02:01.687 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/dispatcher/supervisor.py", line 227, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi140
2023-07-25T14:02:01.688 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-25T14:02:01.688 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-25T14:02:06.693 DEBUG:teuthology.provision.fog.smithi002:FOG request: GET /task/active None
2023-07-25T14:02:06.696 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:02:06.745 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-25T14:02:06.778 INFO:teuthology.nuke:Checking targets against current locks
2023-07-25T14:02:11.787 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:02:11.806 ERROR:teuthology.orchestra.connection:Error authenticating with smithi002.front.sepia.ceph.com: Authentication failed.
2023-07-25T14:02:11.807 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-07-25T14:02:11.807 DEBUG:teuthology.orchestra.run.smithi002:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-07-25T14:02:11.808 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:02:21.822 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:02:21.822 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:02:21.846 DEBUG:teuthology.nuke:shortname: smithi002
2023-07-25T14:02:21.847 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-25T14:02:36.865 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:02:36.865 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:02:36.892 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-07-25_11:25:54-fs-wip-vshankar-testing-20230725.053049-testing-default-smithi/7350653', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-25 13:58:19.485193', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9O1HgNQnCZEi9u4Q2Pmi8SBe8qXX6JbdLahRhcm70z99c68Gsm0zYypZclolIq8USRzi1pMgD9vIuPhC1dndgwD6FUk9f0I56pS9Zsv+SJpdG6IRvd+qYS1/gzBI21C6ldIooYAId0uFzDWxLPSPlmQVfUlgM47pOW5rgDisd2F01z+X8CYdJ45xpY8au1lyXovEUGuVKfNJ8dhe751cEpN9Ltd3nI7xo7Fbp1nqaJDakFmLiLUSRs+HWOeTB+Py6WDTbOHLm3fs7omG2rrzvaTTM+HQJZQX6iGk628EZN6dcy82m3mdzSyNoekedvZ8wTpCA3EuHjoQechy9tbXvpHwhyDUXKqQG/zvhJI1TOtgaM/wLaMqmVomqgx/MVSl7IurNJwe00ByaFlFCMcc87IjWm78hd5aK0rt6H8LQHK6EATUS0M8HiizotETkz1vsSXHfWWgPH8OcvpL+BzAu+Gh2EHp/dC47Uj0wXyVyB4lGVAhatmP6QNzqrQV7Pr8='}
2023-07-25T14:02:41.899 DEBUG:teuthology.nuke:shortname: smithi087
2023-07-25T14:02:41.899 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-25T14:02:51.909 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:02:51.910 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:02:51.936 DEBUG:teuthology.nuke:shortname: smithi140
2023-07-25T14:02:51.937 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-25T14:02:53.140 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi087.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-07-25_11:25:54-fs-wip-vshankar-testing-20230725.053049-testing-default-smithi/7350653', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-25 13:58:19.488066', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='}
2023-07-25T14:02:53.148 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:02:53.148 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:02:53.209 INFO:teuthology.orchestra.console:Power off smithi002
2023-07-25T14:02:53.209 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-25T14:02:53.222 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi140.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-07-25_11:25:54-fs-wip-vshankar-testing-20230725.053049-testing-default-smithi/7350653', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-25 13:58:19.486855', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDN1Ok85h7Po70ZZPsMuQ9oNzHm6Mer58mEtLakhR92wrOCK2I681t/XHPfAi3BXVR5WOWUu5MAaux3vMYXySvVgZEkU7HdrnZdlzYmaRCzsPXxqOAP7NgSh+kSj0saRUjh62OKk8FaL9DpolW76S/Osc5ftBEq3QIc16fFBx5R4pwvIjZKA6aCJ54uBulxtWOxmDZehSk5cMiAxFa+MjwruqGW07iwH0lZ7IgvyFqrTfO6VQq2CUWQnliO23hlZnHsXG+sy1qZD4ZgdK2dB86HiqhuTS80Zmq7ZBVHost0i+nj3vm5n9j5+Jgo+yipp+Mq7XbQsa/xGQRlGpzGtK322w38qnLPxvpn/LUHEOmSfh7JL9jS1bD/HFVbURccmgpSyS0GUpoCY1EzqIKJf0Jv67IEXxOrKwceVSVSU83Z356SJkckNCteihIfQKeBoR39wJyaBpSk3xQfbYIv+YtrKsk4hv/guYUndgOJEx9rfWDm9AoKXe8wyIzD8/qmlts='}
2023-07-25T14:02:53.235 DEBUG:teuthology.orchestra.console:power off output: Chassis Power Control: Down/Off
2023-07-25T14:02:53.235 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:02:53.260 INFO:teuthology.orchestra.console:Power off smithi087
2023-07-25T14:02:53.261 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-25T14:02:53.275 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-25T14:02:53.283 DEBUG:teuthology.orchestra.console:power off output: Chassis Power Control: Down/Off
2023-07-25T14:02:53.283 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:02:53.302 INFO:teuthology.orchestra.console:Power off smithi140
2023-07-25T14:02:53.303 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-25T14:02:53.314 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-25T14:02:53.329 DEBUG:teuthology.orchestra.console:power off output: Chassis Power Control: Down/Off
2023-07-25T14:02:53.329 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:02:53.375 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-25T14:02:54.161 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:02:54.162 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:02:55.167 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:02:55.168 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:02:56.173 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:02:56.174 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:02:57.180 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:02:57.181 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:02:57.277 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:02:57.316 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:02:57.377 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:02:57.491 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-25T14:02:58.195 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:02:58.196 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:02:58.392 DEBUG:teuthology.orchestra.console:check power output: 
2023-07-25T14:02:58.427 DEBUG:teuthology.orchestra.console:check power output: 
2023-07-25T14:02:59.202 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:02:59.202 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:03:00.208 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:03:00.209 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-25T14:03:01.492 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:03:01.605 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-25T14:03:02.393 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:03:02.428 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:03:03.505 DEBUG:teuthology.orchestra.console:check power output: 
2023-07-25T14:03:03.539 DEBUG:teuthology.orchestra.console:check power output: 
2023-07-25T14:03:05.606 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:03:05.720 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off
2023-07-25T14:03:05.822 INFO:teuthology.orchestra.console:Power off for smithi140 completed
2023-07-25T14:03:10.928 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:03:10.938 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:03:15.982 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off
2023-07-25T14:03:16.021 INFO:teuthology.lock.ops:unlocked: smithi140.front.sepia.ceph.com
2023-07-25T14:03:16.084 INFO:teuthology.orchestra.console:Power off for smithi087 completed
2023-07-25T14:03:21.189 DEBUG:teuthology.orchestra.console:check power output: 
2023-07-25T14:03:26.221 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:03:26.261 INFO:teuthology.lock.ops:unlocked: smithi087.front.sepia.ceph.com
2023-07-25T14:03:27.331 DEBUG:teuthology.orchestra.console:check power output: 
2023-07-25T14:03:31.332 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-25T14:03:31.443 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off
2023-07-25T14:03:31.545 INFO:teuthology.orchestra.console:Power off for smithi002 completed
2023-07-25T14:03:40.031 INFO:teuthology.lock.ops:unlocked: smithi002.front.sepia.ceph.com
2023-07-25T14:03:40.447 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/dispatcher/supervisor.py", line 57, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/dispatcher/supervisor.py", line 227, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi140