2023-08-09T21:28:55.211 INFO:root:teuthology version: 0.0.1.dev181+g7fda959
2023-08-09T21:28:55.212 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-08-09T21:28:55.349 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f5495321850>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f5496590280>, signals=[15])
2023-08-09T21:28:55.351 INFO:teuthology.lock.ops:Start node 'smithi140.front.sepia.ceph.com' reimaging
2023-08-09T21:28:55.352 INFO:teuthology.lock.ops:Updating [smithi140.front.sepia.ceph.com]: reset os type and version on server
2023-08-09T21:28:55.352 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server
2023-08-09T21:28:55.370 INFO:teuthology.lock.ops:Start node 'smithi174.front.sepia.ceph.com' reimaging
2023-08-09T21:28:55.371 INFO:teuthology.lock.ops:Updating [smithi174.front.sepia.ceph.com]: reset os type and version on server
2023-08-09T21:28:55.371 INFO:teuthology.lock.ops:Updating smithi174.front.sepia.ceph.com on lock server
2023-08-09T21:28:55.373 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /host {"name": "smithi140"}
2023-08-09T21:28:55.424 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /host {"name": "smithi174"}
2023-08-09T21:28:55.447 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-08-09T21:28:55.485 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-08-09T21:28:55.488 DEBUG:teuthology.provision.fog.smithi140:Requesting image smithi_rhel_8.6 (ID 5)
2023-08-09T21:28:55.488 DEBUG:teuthology.provision.fog.smithi140:FOG request: PUT /host/245 {"imageID": 5}
2023-08-09T21:28:55.521 DEBUG:teuthology.provision.fog.smithi174:Requesting image smithi_rhel_8.6 (ID 5)
2023-08-09T21:28:55.522 DEBUG:teuthology.provision.fog.smithi174:FOG request: PUT /host/279 {"imageID": 5}
2023-08-09T21:28:55.530 INFO:teuthology.provision.fog.smithi140:Scheduling deploy of rhel 8.6
2023-08-09T21:28:55.530 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-09T21:28:55.567 INFO:teuthology.provision.fog.smithi174:Scheduling deploy of rhel 8.6
2023-08-09T21:28:55.567 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-08-09T21:28:55.583 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /tasktype {"name": "deploy"}
2023-08-09T21:28:55.612 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /tasktype {"name": "deploy"}
2023-08-09T21:28:55.619 DEBUG:teuthology.provision.fog.smithi140:FOG request: POST /host/245/task {"taskTypeID": 1}
2023-08-09T21:28:55.643 DEBUG:teuthology.provision.fog.smithi174:FOG request: POST /host/279/task {"taskTypeID": 1}
2023-08-09T21:28:55.662 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-09T21:28:55.708 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-08-09T21:28:55.761 INFO:teuthology.orchestra.console.smithi140:Power off
2023-08-09T21:28:55.764 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-08-09T21:28:55.786 DEBUG:teuthology.orchestra.console.smithi140:power off output: Chassis Power Control: Down/Off
2023-08-09T21:28:55.787 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-09T21:28:55.807 INFO:teuthology.orchestra.console.smithi174:Power off
2023-08-09T21:28:55.807 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-08-09T21:28:55.817 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is off
2023-08-09T21:28:55.827 DEBUG:teuthology.orchestra.console.smithi174:power off output: Chassis Power Control: Down/Off
2023-08-09T21:28:55.827 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-09T21:28:55.849 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is off
2023-08-09T21:28:55.918 INFO:teuthology.orchestra.console.smithi140:Power off completed
2023-08-09T21:28:55.951 INFO:teuthology.orchestra.console.smithi174:Power off completed
2023-08-09T21:28:56.020 INFO:teuthology.orchestra.console.smithi140:Power on
2023-08-09T21:28:56.020 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-08-09T21:28:56.043 DEBUG:teuthology.orchestra.console.smithi140:power on output: Chassis Power Control: Up/On
2023-08-09T21:28:56.043 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-09T21:28:56.053 INFO:teuthology.orchestra.console.smithi174:Power on
2023-08-09T21:28:56.053 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-08-09T21:28:56.066 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is off
2023-08-09T21:28:56.075 DEBUG:teuthology.orchestra.console.smithi174:power on output: Chassis Power Control: Up/On
2023-08-09T21:28:56.075 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-09T21:28:56.096 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is off
2023-08-09T21:29:00.068 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-09T21:29:00.098 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-09T21:29:00.177 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is on
2023-08-09T21:29:00.208 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is off
2023-08-09T21:29:00.279 INFO:teuthology.orchestra.console.smithi140:Power on completed
2023-08-09T21:29:00.380 INFO:teuthology.provision.fog.smithi140:Waiting for deploy to finish
2023-08-09T21:29:00.380 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-09T21:29:04.212 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-09T21:29:04.322 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on
2023-08-09T21:29:04.423 INFO:teuthology.orchestra.console.smithi174:Power on completed
2023-08-09T21:29:04.524 INFO:teuthology.provision.fog.smithi174:Waiting for deploy to finish
2023-08-09T21:29:04.525 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-08-09T21:29:15.448 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-09T21:29:19.588 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-08-09T21:29:30.516 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-09T21:29:34.661 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-08-09T21:29:45.588 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-09T21:29:49.736 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-08-09T21:30:00.660 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-09T21:30:04.801 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-08-09T21:30:15.724 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-09T21:30:19.860 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-08-09T21:30:30.788 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-09T21:30:34.916 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-08-09T21:30:45.844 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-09T21:30:49.971 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-08-09T21:31:00.903 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-09T21:31:05.036 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-08-09T21:31:15.960 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-09T21:31:20.092 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-08-09T21:31:31.016 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-09T21:31:35.148 DEBUG:teuthology.provision.fog.smithi174:FOG request: GET /task/active None
2023-08-09T21:31:35.194 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-09T21:31:38.256 WARNING:teuthology.provision.fog.smithi174:[Errno None] Unable to connect to port 22 on 172.21.15.174
2023-08-09T21:31:44.260 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-09T21:31:46.061 DEBUG:teuthology.provision.fog.smithi140:FOG request: GET /task/active None
2023-08-09T21:31:46.114 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-09T21:31:46.225 ERROR:teuthology.orchestra.connection:Error authenticating with smithi140.front.sepia.ceph.com: Authentication failed.
2023-08-09T21:31:46.226 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-08-09T21:31:46.227 DEBUG:teuthology.orchestra.run.smithi140:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-08-09T21:31:46.228 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-08-09T21:31:47.233 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-08-09T21:31:47.312 WARNING:teuthology.provision.fog.smithi174:[Errno None] Unable to connect to port 22 on 172.21.15.174
2023-08-09T21:31:53.314 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-09T21:31:56.374 WARNING:teuthology.provision.fog.smithi174:[Errno None] Unable to connect to port 22 on 172.21.15.174
2023-08-09T21:32:02.376 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-09T21:32:05.456 WARNING:teuthology.provision.fog.smithi174:[Errno None] Unable to connect to port 22 on 172.21.15.174
2023-08-09T21:32:11.457 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-09T21:32:14.512 WARNING:teuthology.provision.fog.smithi174:[Errno None] Unable to connect to port 22 on 172.21.15.174
2023-08-09T21:32:20.513 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-09T21:32:23.568 WARNING:teuthology.provision.fog.smithi174:[Errno None] Unable to connect to port 22 on 172.21.15.174
2023-08-09T21:32:29.570 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-09T21:32:32.624 WARNING:teuthology.provision.fog.smithi174:[Errno None] Unable to connect to port 22 on 172.21.15.174
2023-08-09T21:32:38.625 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-09T21:33:38.627 WARNING:teuthology.provision.fog.smithi174:timed out
2023-08-09T21:33:44.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-08-09T21:33:44.960 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-08-09T21:33:44.960 DEBUG:teuthology.orchestra.run.smithi174:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-08-09T21:33:45.252 INFO:teuthology.provision.fog.smithi174:Node is ready
2023-08-09T21:33:45.252 DEBUG:teuthology.orchestra.run.smithi174:> hostname
2023-08-09T21:33:45.269 INFO:teuthology.orchestra.run.smithi174.stdout:smithi174
2023-08-09T21:33:45.270 DEBUG:teuthology.orchestra.run.smithi174:> grep smithi174 /etc/hosts
2023-08-09T21:33:45.327 INFO:teuthology.orchestra.run.smithi174.stdout:172.21.15.174 smithi174.front.sepia.ceph.com smithi174
2023-08-09T21:33:45.328 DEBUG:teuthology.orchestra.run.smithi174:> sudo hostname smithi174
2023-08-09T21:33:45.448 DEBUG:teuthology.orchestra.run.smithi174:> sudo sed -i -e 's/smithi174/smithi174/g' /etc/hosts
2023-08-09T21:33:45.477 DEBUG:teuthology.orchestra.run.smithi174:> sudo sed -i -e 's/172.21.15.174/172.21.15.174/g' /etc/hosts
2023-08-09T21:33:45.546 DEBUG:teuthology.orchestra.run.smithi174:> sudo sed -i -e 's/smithi174/smithi174/g' /etc/hostname
2023-08-09T21:33:45.614 DEBUG:teuthology.orchestra.run.smithi174:> sudo hostname smithi174
2023-08-09T21:33:45.681 DEBUG:teuthology.orchestra.run.smithi174:> cat /etc/os-release
2023-08-09T21:33:45.738 INFO:teuthology.orchestra.run.smithi174.stdout:NAME="Red Hat Enterprise Linux"
2023-08-09T21:33:45.738 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION="8.6 (Ootpa)"
2023-08-09T21:33:45.738 INFO:teuthology.orchestra.run.smithi174.stdout:ID="rhel"
2023-08-09T21:33:45.738 INFO:teuthology.orchestra.run.smithi174.stdout:ID_LIKE="fedora"
2023-08-09T21:33:45.739 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_ID="8.6"
2023-08-09T21:33:45.739 INFO:teuthology.orchestra.run.smithi174.stdout:PLATFORM_ID="platform:el8"
2023-08-09T21:33:45.739 INFO:teuthology.orchestra.run.smithi174.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-08-09T21:33:45.739 INFO:teuthology.orchestra.run.smithi174.stdout:ANSI_COLOR="0;31"
2023-08-09T21:33:45.739 INFO:teuthology.orchestra.run.smithi174.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-08-09T21:33:45.739 INFO:teuthology.orchestra.run.smithi174.stdout:HOME_URL="https://www.redhat.com/"
2023-08-09T21:33:45.739 INFO:teuthology.orchestra.run.smithi174.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-08-09T21:33:45.740 INFO:teuthology.orchestra.run.smithi174.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-08-09T21:33:45.740 INFO:teuthology.orchestra.run.smithi174.stdout:
2023-08-09T21:33:45.740 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-08-09T21:33:45.740 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-08-09T21:33:45.740 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-08-09T21:33:45.740 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-08-09T21:33:45.741 INFO:teuthology.provision.fog.smithi174:Deploy complete!
2023-08-09T21:33:45.742 DEBUG:teuthology.parallel:result is None
2023-08-09T21:41:46.281 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/provision/fog.py", line 90, in create
    self._wait_for_ready()
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/provision/fog.py", line 299, in _wait_for_ready
    self.remote.run(args=cmd, timeout=600)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/orchestra/remote.py", line 522, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/orchestra/run.py", line 450, in run
    r.execute()
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/orchestra/run.py", line 97, in execute
    self.client.exec_command(self.command, timeout=self.timeout)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 510, in exec_command
    chan = self._transport.open_session(timeout=timeout)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 919, in open_session
    return self.open_channel(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1054, in open_channel
    raise SSHException("Timeout opening channel.")
paramiko.ssh_exception.SSHException: Timeout opening channel.
2023-08-09T21:41:46.283 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-08-09T21:41:46.436 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=75fef98f7ccc4fb1816f5aa1a39146ef
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/provision/fog.py", line 90, in create
    self._wait_for_ready()
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/provision/fog.py", line 299, in _wait_for_ready
    self.remote.run(args=cmd, timeout=600)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/orchestra/remote.py", line 522, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/orchestra/run.py", line 450, in run
    r.execute()
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/orchestra/run.py", line 97, in execute
    self.client.exec_command(self.command, timeout=self.timeout)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 510, in exec_command
    chan = self._transport.open_session(timeout=timeout)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 919, in open_session
    return self.open_channel(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1054, in open_channel
    raise SSHException("Timeout opening channel.")
paramiko.ssh_exception.SSHException: Timeout opening channel.
2023-08-09T21:41:46.440 INFO:teuthology.nuke:Checking targets against current locks
2023-08-09T21:41:46.508 DEBUG:teuthology.nuke:shortname: smithi140
2023-08-09T21:41:46.509 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-08-09T21:41:46.525 DEBUG:teuthology.nuke:shortname: smithi174
2023-08-09T21:41:46.525 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-08-09T21:41:46.532 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi140.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2023-08-05_14:23:03-upgrade:pacific-x-reef-distro-default-smithi/7361794', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-09 21:28:54.384593', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpScFukdwDzr/lh+nEN7BOBMlreYnsIrOTvHYYBRTgB+90bqIVuUuzzRpttLKt5JGcg2TQrsinhpW+yLxMYENV9SIrS6PrCwVFddnaD0H1MqpgW2m3D/pfAPrSrPtId2QGLJ3Pkk0ex6lJ4uUN+AYxnJpGEcg7NI6+MGEpcH/Sd723eleeXxJXaispTNkEk8uV+Hs4eKYxyIheUc/G1gQzJaJ/r7NAyFm0nElhrkztId4HmRHwqF9fPmEqRt4i6I8Twc1cCdip8jW/sS/zu8Z7+VeaDItp241fxAMm/Xa5FQqHTIqmnlQn1r2LYJdR0/lAzXBGE/duJK0Dwsicybc5p+gaTc8Dt3z1ViyUq79mr/4GtsdfdslK/gwmNDQzPKf9G3QeJlYkNFG0YBW76h1nRIVC4uEa9uTh2EoHzjm+WSxB4fKUv8dDx9Z60N6b4AKYfIm25V03M2gMjRNkF7NXV8F/f1kreud5VBZVIrl61r9JrEm/mziLJ0wUCjy5Tis='}
2023-08-09T21:41:46.547 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2023-08-05_14:23:03-upgrade:pacific-x-reef-distro-default-smithi/7361794', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-09 21:28:54.382955', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC64tb0qz6J9iCh9PaZIM0k9N8tbSXV+R/FNl0zTlPWInIsBMc/1GJQwIES/Kq3qmdOc5/2B2rL73htgzr9oT9kLMUMHTlrwQ3lnT6p1ZjGBhCxWRjCoO2nVrRPMbG9xcX3eXDyYkHboLm020z0W9ZN4C7/ujDorkdCnyrPAE/xYF0wcIJhAF3/VNq9LEsQqjKaN0AjssRjf63t7RQoX4FbmV5w60dfQpyVnoJz3EeLtgO3cSjNPFBAcNtfisGijJcMejF8Y5TuK8ghswOi93A3j/gQujUOJ9p9yACrZPH6a0zF9wFStKFdB3ZmCi9n0jRBsP53cq9+bOI9cR5VP93AIBRh6Nh5iXpTIPBmcMGDCILxrjIrzs4yjnrl/+r+q/yICjXmUF82N43dapxoBd93YdtPwmm2tGtw2C4TS++wNTNc8KDtZMgBwBI5FS2nZBomx0wL+ixKc+WgVPC0MwiOaPDAg0HwRCgLtbHQCsl4cVMBD+ITVAtqMQPE7NJK2u8='}
2023-08-09T21:41:46.580 INFO:teuthology.orchestra.console.smithi140:Power off
2023-08-09T21:41:46.581 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-08-09T21:41:46.597 INFO:teuthology.orchestra.console.smithi174:Power off
2023-08-09T21:41:46.598 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-08-09T21:41:46.607 DEBUG:teuthology.orchestra.console.smithi140:power off output: Chassis Power Control: Down/Off
2023-08-09T21:41:46.608 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-09T21:41:46.619 DEBUG:teuthology.orchestra.console.smithi174:power off output: Chassis Power Control: Down/Off
2023-08-09T21:41:46.620 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-09T21:41:46.632 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is on
2023-08-09T21:41:46.645 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on
2023-08-09T21:41:50.634 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-09T21:41:50.646 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-09T21:41:50.748 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is on
2023-08-09T21:41:50.756 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on
2023-08-09T21:41:54.750 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-09T21:41:54.758 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-09T21:41:54.860 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is off
2023-08-09T21:41:54.868 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on
2023-08-09T21:41:54.961 INFO:teuthology.orchestra.console.smithi140:Power off completed
2023-08-09T21:41:55.120 INFO:teuthology.lock.ops:unlocked: smithi140.front.sepia.ceph.com
2023-08-09T21:41:58.869 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-08-09T21:41:58.985 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is off
2023-08-09T21:41:59.088 INFO:teuthology.orchestra.console.smithi174:Power off completed
2023-08-09T21:41:59.246 INFO:teuthology.lock.ops:unlocked: smithi174.front.sepia.ceph.com
2023-08-09T21:41:59.495 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/provision/fog.py", line 90, in create
    self._wait_for_ready()
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/provision/fog.py", line 299, in _wait_for_ready
    self.remote.run(args=cmd, timeout=600)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/orchestra/remote.py", line 522, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/orchestra/run.py", line 450, in run
    r.execute()
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/teuthology/orchestra/run.py", line 97, in execute
    self.client.exec_command(self.command, timeout=self.timeout)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 510, in exec_command
    chan = self._transport.open_session(timeout=timeout)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 919, in open_session
    return self.open_channel(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1054, in open_channel
    raise SSHException("Timeout opening channel.")
paramiko.ssh_exception.SSHException: Timeout opening channel.