2023-06-21T01:39:12.134 INFO:root:teuthology version: 0.0.1.dev128+g076bbeb
2023-06-21T01:39:12.136 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-06-21T01:39:12.136 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-06-21T01:39:12.288 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f451c1276d0>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f451d3458b0>, signals=[15])
2023-06-21T01:39:12.290 INFO:teuthology.lock.ops:Start node 'smithi031.front.sepia.ceph.com' reimaging
2023-06-21T01:39:12.291 INFO:teuthology.lock.ops:Updating [smithi031.front.sepia.ceph.com]: reset os type and version on server
2023-06-21T01:39:12.292 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server
2023-06-21T01:39:12.314 INFO:teuthology.lock.ops:Start node 'smithi039.front.sepia.ceph.com' reimaging
2023-06-21T01:39:12.314 INFO:teuthology.lock.ops:Updating [smithi039.front.sepia.ceph.com]: reset os type and version on server
2023-06-21T01:39:12.315 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server
2023-06-21T01:39:12.316 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /host {"name": "smithi031"}
2023-06-21T01:39:12.339 DEBUG:teuthology.provision.fog.smithi039:FOG request: GET /host {"name": "smithi039"}
2023-06-21T01:39:12.399 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /image {"name": "smithi_centos_8.stream"}
2023-06-21T01:39:12.405 DEBUG:teuthology.provision.fog.smithi039:FOG request: GET /image {"name": "smithi_centos_8.stream"}
2023-06-21T01:39:12.452 DEBUG:teuthology.provision.fog.smithi031:Requesting image smithi_centos_8.stream (ID 3)
2023-06-21T01:39:12.452 DEBUG:teuthology.provision.fog.smithi031:FOG request: PUT /host/136 {"imageID": 3}
2023-06-21T01:39:12.461 DEBUG:teuthology.provision.fog.smithi039:Requesting image smithi_centos_8.stream (ID 3)
2023-06-21T01:39:12.461 DEBUG:teuthology.provision.fog.smithi039:FOG request: PUT /host/144 {"imageID": 3}
2023-06-21T01:39:12.494 INFO:teuthology.provision.fog.smithi031:Scheduling deploy of centos 8.stream
2023-06-21T01:39:12.495 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-06-21T01:39:12.508 INFO:teuthology.provision.fog.smithi039:Scheduling deploy of centos 8.stream
2023-06-21T01:39:12.509 DEBUG:teuthology.provision.fog.smithi039:FOG request: GET /task/active None
2023-06-21T01:39:12.672 DEBUG:teuthology.provision.fog.smithi039:FOG request: GET /tasktype {"name": "deploy"}
2023-06-21T01:39:12.677 DEBUG:teuthology.provision.fog.smithi031:Canceling deploy task with ID 332400
2023-06-21T01:39:12.677 DEBUG:teuthology.provision.fog.smithi031:FOG request: DELETE /task/cancel {"id": 332400}
2023-06-21T01:39:12.714 DEBUG:teuthology.provision.fog.smithi039:FOG request: POST /host/144/task {"taskTypeID": 1}
2023-06-21T01:39:12.723 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /tasktype {"name": "deploy"}
2023-06-21T01:39:12.754 DEBUG:teuthology.provision.fog.smithi031:FOG request: POST /host/136/task {"taskTypeID": 1}
2023-06-21T01:39:12.769 DEBUG:teuthology.provision.fog.smithi039:FOG request: GET /task/active None
2023-06-21T01:39:12.812 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-06-21T01:39:12.988 INFO:teuthology.orchestra.console:Power off smithi039
2023-06-21T01:39:12.989 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-06-21T01:39:13.012 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-21T01:39:13.038 INFO:teuthology.orchestra.console:Power off smithi031
2023-06-21T01:39:13.039 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-06-21T01:39:13.062 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-21T01:39:13.149 INFO:teuthology.orchestra.console:Power off for smithi039 completed
2023-06-21T01:39:13.185 INFO:teuthology.orchestra.console:Power off for smithi031 completed
2023-06-21T01:39:13.251 INFO:teuthology.orchestra.console:Power on smithi039
2023-06-21T01:39:13.251 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-06-21T01:39:13.272 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-21T01:39:13.286 INFO:teuthology.orchestra.console:Power on smithi031
2023-06-21T01:39:13.287 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-06-21T01:39:13.307 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-21T01:39:17.300 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-21T01:39:17.331 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-21T01:39:21.412 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-21T01:39:21.443 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-21T01:39:21.624 INFO:teuthology.orchestra.console:Power on for smithi039 completed
2023-06-21T01:39:21.655 INFO:teuthology.orchestra.console:Power on for smithi031 completed
2023-06-21T01:39:21.726 INFO:teuthology.provision.fog.smithi039:Waiting for deploy to finish
2023-06-21T01:39:21.726 DEBUG:teuthology.provision.fog.smithi039:FOG request: GET /task/active None
2023-06-21T01:39:21.756 INFO:teuthology.provision.fog.smithi031:Waiting for deploy to finish
2023-06-21T01:39:21.756 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-06-21T01:39:36.901 DEBUG:teuthology.provision.fog.smithi039:FOG request: GET /task/active None
2023-06-21T01:39:36.903 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-06-21T01:39:52.081 DEBUG:teuthology.provision.fog.smithi039:FOG request: GET /task/active None
2023-06-21T01:39:52.083 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-06-21T01:40:07.249 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-06-21T01:40:07.251 DEBUG:teuthology.provision.fog.smithi039:FOG request: GET /task/active None
2023-06-21T01:40:22.412 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-06-21T01:40:22.413 DEBUG:teuthology.provision.fog.smithi039:FOG request: GET /task/active None
2023-06-21T01:40:37.564 DEBUG:teuthology.provision.fog.smithi039:FOG request: GET /task/active None
2023-06-21T01:40:37.565 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-06-21T01:40:52.724 DEBUG:teuthology.provision.fog.smithi039:FOG request: GET /task/active None
2023-06-21T01:40:52.726 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-06-21T01:41:07.868 DEBUG:teuthology.provision.fog.smithi039:FOG request: GET /task/active None
2023-06-21T01:41:07.870 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-06-21T01:41:23.012 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-06-21T01:41:23.014 DEBUG:teuthology.provision.fog.smithi039:FOG request: GET /task/active None
2023-06-21T01:41:38.144 DEBUG:teuthology.provision.fog.smithi039:FOG request: GET /task/active None
2023-06-21T01:41:38.145 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-06-21T01:41:53.269 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-06-21T01:41:53.270 DEBUG:teuthology.provision.fog.smithi039:FOG request: GET /task/active None
2023-06-21T01:41:53.375 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-06-21T01:42:08.388 DEBUG:teuthology.provision.fog.smithi039:FOG request: GET /task/active None
2023-06-21T01:42:23.493 DEBUG:teuthology.provision.fog.smithi039:FOG request: GET /task/active None
2023-06-21T01:42:27.956 WARNING:teuthology.provision.fog.smithi031:[Errno None] Unable to connect to port 22 on 172.21.15.31
2023-06-21T01:42:33.960 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-06-21T01:42:38.588 DEBUG:teuthology.provision.fog.smithi039:FOG request: GET /task/active None
2023-06-21T01:42:38.673 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-06-21T01:42:52.276 WARNING:teuthology.provision.fog.smithi031:[Errno None] Unable to connect to port 22 on 172.21.15.31
2023-06-21T01:42:57.136 WARNING:teuthology.provision.fog.smithi039:[Errno None] Unable to connect to port 22 on 172.21.15.39
2023-06-21T01:42:58.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-06-21T01:43:01.332 WARNING:teuthology.provision.fog.smithi031:[Errno None] Unable to connect to port 22 on 172.21.15.31
2023-06-21T01:43:03.139 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-06-21T01:43:06.196 WARNING:teuthology.provision.fog.smithi039:[Errno None] Unable to connect to port 22 on 172.21.15.39
2023-06-21T01:43:07.335 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-06-21T01:43:10.384 WARNING:teuthology.provision.fog.smithi031:[Errno None] Unable to connect to port 22 on 172.21.15.31
2023-06-21T01:43:12.202 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-06-21T01:43:15.280 WARNING:teuthology.provision.fog.smithi039:[Errno None] Unable to connect to port 22 on 172.21.15.39
2023-06-21T01:43:16.387 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-06-21T01:43:16.705 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-06-21T01:43:16.706 DEBUG:teuthology.orchestra.run.smithi031:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-06-21T01:43:21.282 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-06-21T01:43:24.336 WARNING:teuthology.provision.fog.smithi039:[Errno None] Unable to connect to port 22 on 172.21.15.39
2023-06-21T01:43:30.339 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-06-21T01:43:32.019 INFO:teuthology.provision.fog.smithi031:Node is ready
2023-06-21T01:43:32.020 DEBUG:teuthology.orchestra.run.smithi031:> hostname
2023-06-21T01:43:32.080 INFO:teuthology.orchestra.run.smithi031.stdout:smithi031.front.sepia.ceph.com
2023-06-21T01:43:32.081 DEBUG:teuthology.orchestra.run.smithi031:> grep smithi031.front.sepia.ceph.com /etc/hosts
2023-06-21T01:43:32.141 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31 smithi031.front.sepia.ceph.com smithi031
2023-06-21T01:43:32.142 DEBUG:teuthology.orchestra.run.smithi031:> sudo hostname smithi031
2023-06-21T01:43:32.230 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i -e 's/smithi031.front.sepia.ceph.com/smithi031/g' /etc/hosts
2023-06-21T01:43:32.263 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i -e 's/172.21.15.31/172.21.15.31/g' /etc/hosts
2023-06-21T01:43:32.333 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i -e 's/smithi031.front.sepia.ceph.com/smithi031/g' /etc/hostname
2023-06-21T01:43:32.403 DEBUG:teuthology.orchestra.run.smithi031:> sudo hostname smithi031
2023-06-21T01:43:32.473 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release
2023-06-21T01:43:32.532 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="CentOS Stream"
2023-06-21T01:43:32.533 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="8"
2023-06-21T01:43:32.533 INFO:teuthology.orchestra.run.smithi031.stdout:ID="centos"
2023-06-21T01:43:32.533 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="rhel fedora"
2023-06-21T01:43:32.533 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="8"
2023-06-21T01:43:32.533 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el8"
2023-06-21T01:43:32.534 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="CentOS Stream 8"
2023-06-21T01:43:32.537 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31"
2023-06-21T01:43:32.537 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-06-21T01:43:32.537 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://centos.org/"
2023-06-21T01:43:32.538 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-06-21T01:43:32.538 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-06-21T01:43:32.538 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-06-21T01:43:32.540 INFO:teuthology.provision.fog.smithi031:Deploy complete!
2023-06-21T01:43:32.542 DEBUG:teuthology.parallel:result is None
2023-06-21T01:43:33.392 WARNING:teuthology.provision.fog.smithi039:[Errno None] Unable to connect to port 22 on 172.21.15.39
2023-06-21T01:43:39.396 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-06-21T01:43:42.481 WARNING:teuthology.provision.fog.smithi039:[Errno None] Unable to connect to port 22 on 172.21.15.39
2023-06-21T01:43:48.488 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-06-21T01:43:51.568 WARNING:teuthology.provision.fog.smithi039:[Errno None] Unable to connect to port 22 on 172.21.15.39
2023-06-21T01:43:57.572 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-06-21T01:44:57.636 WARNING:teuthology.provision.fog.smithi039:timed out
2023-06-21T01:45:03.645 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-06-21T01:45:03.885 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-06-21T01:45:03.886 DEBUG:teuthology.orchestra.run.smithi039:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-06-21T01:45:04.172 INFO:teuthology.provision.fog.smithi039:Node is ready
2023-06-21T01:45:04.172 DEBUG:teuthology.orchestra.run.smithi039:> hostname
2023-06-21T01:45:04.193 INFO:teuthology.orchestra.run.smithi039.stdout:smithi039.front.sepia.ceph.com
2023-06-21T01:45:04.194 DEBUG:teuthology.orchestra.run.smithi039:> grep smithi039.front.sepia.ceph.com /etc/hosts
2023-06-21T01:45:04.258 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39 smithi039.front.sepia.ceph.com smithi039
2023-06-21T01:45:04.259 DEBUG:teuthology.orchestra.run.smithi039:> sudo hostname smithi039
2023-06-21T01:45:04.356 DEBUG:teuthology.orchestra.run.smithi039:> sudo sed -i -e 's/smithi039.front.sepia.ceph.com/smithi039/g' /etc/hosts
2023-06-21T01:45:04.386 DEBUG:teuthology.orchestra.run.smithi039:> sudo sed -i -e 's/172.21.15.39/172.21.15.39/g' /etc/hosts
2023-06-21T01:45:04.458 DEBUG:teuthology.orchestra.run.smithi039:> sudo sed -i -e 's/smithi039.front.sepia.ceph.com/smithi039/g' /etc/hostname
2023-06-21T01:45:04.530 DEBUG:teuthology.orchestra.run.smithi039:> sudo hostname smithi039
2023-06-21T01:45:04.602 DEBUG:teuthology.orchestra.run.smithi039:> cat /etc/os-release
2023-06-21T01:45:04.668 INFO:teuthology.orchestra.run.smithi039.stdout:NAME="CentOS Stream"
2023-06-21T01:45:04.668 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION="8"
2023-06-21T01:45:04.669 INFO:teuthology.orchestra.run.smithi039.stdout:ID="centos"
2023-06-21T01:45:04.669 INFO:teuthology.orchestra.run.smithi039.stdout:ID_LIKE="rhel fedora"
2023-06-21T01:45:04.669 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_ID="8"
2023-06-21T01:45:04.669 INFO:teuthology.orchestra.run.smithi039.stdout:PLATFORM_ID="platform:el8"
2023-06-21T01:45:04.670 INFO:teuthology.orchestra.run.smithi039.stdout:PRETTY_NAME="CentOS Stream 8"
2023-06-21T01:45:04.670 INFO:teuthology.orchestra.run.smithi039.stdout:ANSI_COLOR="0;31"
2023-06-21T01:45:04.670 INFO:teuthology.orchestra.run.smithi039.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-06-21T01:45:04.671 INFO:teuthology.orchestra.run.smithi039.stdout:HOME_URL="https://centos.org/"
2023-06-21T01:45:04.671 INFO:teuthology.orchestra.run.smithi039.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-06-21T01:45:04.672 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-06-21T01:45:04.673 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-06-21T01:45:04.673 INFO:teuthology.provision.fog.smithi039:Deploy complete!
2023-06-21T01:45:04.674 DEBUG:teuthology.parallel:result is None
2023-06-21T01:45:09.531 ERROR:teuthology.misc:write (smithi031.front.sepia.ceph.com): Connection refused
2023-06-21T01:45:09.532 ERROR:teuthology.misc:write (smithi031.front.sepia.ceph.com): Connection refused
2023-06-21T01:45:09.533 ERROR:teuthology.misc:write (smithi031.front.sepia.ceph.com): Connection refused
2023-06-21T01:45:10.624 INFO:teuthology.lock.ops:Checking smithi031.front.sepia.ceph.com
2023-06-21T01:45:10.625 INFO:teuthology.lock.ops:New key found. Updating...
2023-06-21T01:45:10.655 INFO:teuthology.lock.ops:Checking smithi039.front.sepia.ceph.com
2023-06-21T01:45:10.655 INFO:teuthology.lock.ops:New key found. Updating...
2023-06-21T01:45:10.680 INFO:teuthology.lock.ops:Updating [smithi031.front.sepia.ceph.com]: set os type and version on server
2023-06-21T01:45:10.681 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com'
2023-06-21T01:45:10.682 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-06-21T01:45:10.809 ERROR:teuthology.orchestra.connection:Error authenticating with smithi031.front.sepia.ceph.com: Authentication failed.
2023-06-21T01:45:11.933 ERROR:teuthology.orchestra.connection:Error authenticating with smithi031.front.sepia.ceph.com: Authentication failed.
2023-06-21T01:45:13.066 ERROR:teuthology.orchestra.connection:Error authenticating with smithi031.front.sepia.ceph.com: Authentication failed.
2023-06-21T01:45:14.189 ERROR:teuthology.orchestra.connection:Error authenticating with smithi031.front.sepia.ceph.com: Authentication failed.
2023-06-21T01:45:15.312 ERROR:teuthology.orchestra.connection:Error authenticating with smithi031.front.sepia.ceph.com: Authentication failed.
2023-06-21T01:45:16.432 ERROR:teuthology.orchestra.connection:Error authenticating with smithi031.front.sepia.ceph.com: Authentication failed.
2023-06-21T01:45:17.558 ERROR:teuthology.orchestra.connection:Error authenticating with smithi031.front.sepia.ceph.com: Authentication failed.
2023-06-21T01:45:18.689 ERROR:teuthology.orchestra.connection:Error authenticating with smithi031.front.sepia.ceph.com: Authentication failed.
2023-06-21T01:45:19.822 ERROR:teuthology.orchestra.connection:Error authenticating with smithi031.front.sepia.ceph.com: Authentication failed.
2023-06-21T01:45:20.960 ERROR:teuthology.orchestra.connection:Error authenticating with smithi031.front.sepia.ceph.com: Authentication failed.
2023-06-21T01:45:20.960 DEBUG:teuthology.orchestra.remote:'connect to smithi031.front.sepia.ceph.com' reached maximum tries (11) after waiting for 10 seconds
2023-06-21T01:45:20.961 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/dispatcher/supervisor.py", line 227, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/lock/ops.py", line 327, in reimage_machines
    update_nodes(reimaged)
  File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/lock/ops.py", line 38, in update_nodes
    inventory_info = remote.inventory_info
  File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 679, in inventory_info
    node['arch'] = self.arch
  File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 335, in arch
    self._arch = self.sh('uname -m').strip()
  File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 96, in sh
    proc = self.run(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi031
2023-06-21T01:45:20.961 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-06-21T01:45:20.962 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-06-21T01:45:21.205 INFO:teuthology.nuke:Checking targets against current locks
2023-06-21T01:45:21.264 DEBUG:teuthology.nuke:shortname: smithi031
2023-06-21T01:45:21.264 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-06-21T01:45:21.299 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2023-06-21_01:27:59-orch:cephadm-wip-adk4-testing-2023-06-20-1932-distro-default-smithi/7309343', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-21 01:43:42.373001', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCe0Zv8Vkzr942CvXQtZB7zrUTcy+q/fzPv8rTdMi2+Lgn0GIh2ZcofYARD4CvBgeZiNFRrJoIXCumV4Q6i+NLUyVHldWi7BkDsOrS4xzgn9w9P9iUVBomvam4ck1pi/F2Z06ZDgdDMn5ubuVxXIFgIYGPvZa8ypkiDcABxRsWMqsQ/GGqaQiOLOR1KITdPBtbyL8Tcv/DFMzRFGQPkrgys5rXMVlc05pEmtBxpAYVtvuj7oTTFOQqj04EOF77UOOXVswYQQXs6vzsmcvcSIpgNj9ualDL2lxNOiWsWMb2JsLxRySp3wopTpyu0VRs4rVjtLkwhB7Rbk1QHDXNa6QY50n1g8ZWFYmcbOy7A7AAnXNqwD1ytwnSo8eRv1gs4TsSJ765F4hleAfc9ZEp1BWeDnQ7iVKoibKsZjDrRoHqA2TK772/isXG9YcSTvo3faq2+nVv6Gqj+vs4PJ5vR7Hs8OEVmDTC0lbefk7dKpe5sgbVCEM3zu4/rVfj2rRQiT9M='}
2023-06-21T01:45:21.317 DEBUG:teuthology.nuke:shortname: smithi039
2023-06-21T01:45:21.317 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-06-21T01:45:21.349 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi039.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2023-06-21_01:27:59-orch:cephadm-wip-adk4-testing-2023-06-20-1932-distro-default-smithi/7309342', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-21 01:39:11.347137', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNe/svXIEgk6ut9sTgdxse6IlmLk6ac9zTAHqMM+6sv4LqhvSdaAwh/BizMOKFilakSYUmmqGDUczThC9nYwiuxXp+setwNwoJQeE94pHKfjRDz9wTeqlOBcj+rovtxvdqGrXeEvyO60rlZcfrFGH9NwZ8ckgkHwOsXU9eVWUseIH/wZGus1dltZcFoC+LEkGhzy03EGBZ3WR1gBBzCivhf8Wj8lIbclbKLgP4kSFs+qwnBqifHlCiLgOl4+s06A9CkGPzsG+O5QRxoH1S8wo3K3NMzkDsswVf6UnJG4YVuQfq7hgP1bESfA3MbctNuFEZz0olUk499O/d3NwNGp+9PlCyXwSfPndu2TiMpQlRcZWOGVd6FpchMgR8Yxx3PTh9bBnwnEirMoPybL6mnBRS1Bipv/rkWaD56kh1/wJDHYoqdgcmv1N5BXLMsJqwKVFimvksKeET98eEkob/r5QKGwmFtODE4LmcFMHSqfy3eXDxb50uMWEyO2gtiSPMLQk='}
2023-06-21T01:45:21.398 INFO:teuthology.orchestra.console:Power off smithi031
2023-06-21T01:45:21.399 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-06-21T01:45:21.440 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-21T01:45:21.483 INFO:teuthology.orchestra.console:Power off smithi039
2023-06-21T01:45:21.484 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-06-21T01:45:21.516 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-21T01:45:25.507 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-21T01:45:25.553 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-21T01:45:29.623 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-21T01:45:29.673 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-06-21T01:45:29.845 INFO:teuthology.orchestra.console:Power off for smithi031 completed
2023-06-21T01:45:29.892 INFO:teuthology.orchestra.console:Power off for smithi039 completed
2023-06-21T01:45:30.038 INFO:teuthology.lock.ops:unlocked: smithi031.front.sepia.ceph.com
2023-06-21T01:45:30.074 INFO:teuthology.lock.ops:unlocked: smithi039.front.sepia.ceph.com
2023-06-21T01:45:30.366 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/dispatcher/supervisor.py", line 57, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/dispatcher/supervisor.py", line 227, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/lock/ops.py", line 327, in reimage_machines
    update_nodes(reimaged)
  File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/lock/ops.py", line 38, in update_nodes
    inventory_info = remote.inventory_info
  File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 679, in inventory_info
    node['arch'] = self.arch
  File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 335, in arch
    self._arch = self.sh('uname -m').strip()
  File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 96, in sh
    proc = self.run(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi031