2023-07-14T17:46:55.136 INFO:root:teuthology version: 0.0.1.dev145+gaa6a80e
2023-07-14T17:46:55.138 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-14T17:46:55.138 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-14T17:46:55.349 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7efd6537e520>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7efd665a08b0>, signals=[15])
2023-07-14T17:46:55.350 INFO:teuthology.lock.ops:Start node 'smithi019.front.sepia.ceph.com' reimaging
2023-07-14T17:46:55.350 INFO:teuthology.lock.ops:Updating [smithi019.front.sepia.ceph.com]: reset os type and version on server
2023-07-14T17:46:55.351 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server
2023-07-14T17:46:55.376 INFO:teuthology.lock.ops:Start node 'smithi031.front.sepia.ceph.com' reimaging
2023-07-14T17:46:55.376 INFO:teuthology.lock.ops:Updating [smithi031.front.sepia.ceph.com]: reset os type and version on server
2023-07-14T17:46:55.377 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server
2023-07-14T17:46:55.381 DEBUG:teuthology.provision.fog.smithi019:FOG request: GET /host {"name": "smithi019"}
2023-07-14T17:46:58.697 INFO:teuthology.lock.ops:Start node 'smithi202.front.sepia.ceph.com' reimaging
2023-07-14T17:46:58.698 INFO:teuthology.lock.ops:Updating [smithi202.front.sepia.ceph.com]: reset os type and version on server
2023-07-14T17:46:58.699 INFO:teuthology.lock.ops:Updating smithi202.front.sepia.ceph.com on lock server
2023-07-14T17:46:58.704 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /host {"name": "smithi031"}
2023-07-14T17:47:21.506 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /host {"name": "smithi202"}
2023-07-14T17:47:29.191 DEBUG:teuthology.provision.fog.smithi019:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-07-14T17:47:39.206 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-07-14T17:47:44.214 DEBUG:teuthology.provision.fog.smithi019:Requesting image smithi_rhel_8.6 (ID 5)
2023-07-14T17:47:44.214 DEBUG:teuthology.provision.fog.smithi019:FOG request: PUT /host/125 {"imageID": 5}
2023-07-14T17:47:49.222 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-07-14T17:47:49.284 DEBUG:teuthology.provision.fog.smithi202:Requesting image smithi_rhel_8.6 (ID 5)
2023-07-14T17:47:49.285 DEBUG:teuthology.provision.fog.smithi202:FOG request: PUT /host/306 {"imageID": 5}
2023-07-14T17:47:49.286 DEBUG:teuthology.provision.fog.smithi031:Requesting image smithi_rhel_8.6 (ID 5)
2023-07-14T17:47:49.286 DEBUG:teuthology.provision.fog.smithi031:FOG request: PUT /host/136 {"imageID": 5}
2023-07-14T17:47:59.297 INFO:teuthology.provision.fog.smithi019:Scheduling deploy of rhel 8.6
2023-07-14T17:47:59.298 DEBUG:teuthology.provision.fog.smithi019:FOG request: GET /task/active None
2023-07-14T17:48:04.306 INFO:teuthology.provision.fog.smithi202:Scheduling deploy of rhel 8.6
2023-07-14T17:48:04.306 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:48:04.417 INFO:teuthology.provision.fog.smithi031:Scheduling deploy of rhel 8.6
2023-07-14T17:48:04.417 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-07-14T17:48:04.431 DEBUG:teuthology.provision.fog.smithi019:FOG request: GET /tasktype {"name": "deploy"}
2023-07-14T17:48:04.434 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /tasktype {"name": "deploy"}
2023-07-14T17:48:04.481 DEBUG:teuthology.provision.fog.smithi202:FOG request: POST /host/306/task {"taskTypeID": 1}
2023-07-14T17:48:04.485 DEBUG:teuthology.provision.fog.smithi019:FOG request: POST /host/125/task {"taskTypeID": 1}
2023-07-14T17:48:04.492 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /tasktype {"name": "deploy"}
2023-07-14T17:48:04.534 DEBUG:teuthology.provision.fog.smithi031:FOG request: POST /host/136/task {"taskTypeID": 1}
2023-07-14T17:48:04.557 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:48:04.559 DEBUG:teuthology.provision.fog.smithi019:FOG request: GET /task/active None
2023-07-14T17:48:04.605 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-07-14T17:48:19.679 INFO:teuthology.orchestra.console:Power off smithi202
2023-07-14T17:48:19.680 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi202.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-14T17:48:19.696 INFO:teuthology.orchestra.console:Power off smithi019
2023-07-14T17:48:19.697 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-14T17:48:19.711 INFO:teuthology.orchestra.console:Power off smithi031
2023-07-14T17:48:19.711 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-14T17:48:19.721 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:48:19.730 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi202.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:48:19.741 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:48:19.854 INFO:teuthology.orchestra.console:Power off for smithi202 completed
2023-07-14T17:48:19.955 INFO:teuthology.orchestra.console:Power on smithi202
2023-07-14T17:48:19.956 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi202.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-14T17:48:19.980 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi202.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:48:23.752 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:48:23.763 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:48:24.003 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi202.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:48:28.865 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:48:28.875 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:48:29.115 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi202.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:48:33.978 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:48:33.989 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:48:34.226 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi202.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:48:34.437 INFO:teuthology.orchestra.console:Power on for smithi202 completed
2023-07-14T17:48:34.538 INFO:teuthology.provision.fog.smithi202:Waiting for deploy to finish
2023-07-14T17:48:34.538 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:48:39.646 INFO:teuthology.orchestra.console:Power off for smithi019 completed
2023-07-14T17:48:39.747 INFO:teuthology.orchestra.console:Power on smithi019
2023-07-14T17:48:39.748 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-14T17:48:43.548 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:48:44.778 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:48:48.660 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:48:49.791 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:48:53.771 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:48:54.613 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:48:54.903 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:48:58.884 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:49:00.015 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:49:03.996 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:49:05.128 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:49:09.109 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:49:09.679 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:49:14.685 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:49:18.696 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:49:19.357 INFO:teuthology.orchestra.console:Power off for smithi031 completed
2023-07-14T17:49:19.458 INFO:teuthology.orchestra.console:Power on smithi031
2023-07-14T17:49:19.458 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-14T17:49:19.482 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:49:19.796 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:49:23.509 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:49:24.907 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:49:28.620 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:49:29.763 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:49:34.769 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:49:34.779 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:49:39.882 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:49:39.890 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:49:44.994 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:49:45.002 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:49:49.848 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:49:50.106 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:49:50.115 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:49:55.219 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:49:55.228 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:50:00.332 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:50:00.342 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:50:05.096 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:50:10.090 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:50:10.098 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:50:10.302 INFO:teuthology.orchestra.console:Power on for smithi019 completed
2023-07-14T17:50:10.403 INFO:teuthology.provision.fog.smithi019:Waiting for deploy to finish
2023-07-14T17:50:10.403 DEBUG:teuthology.provision.fog.smithi019:FOG request: GET /task/active None
2023-07-14T17:50:15.473 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-14T17:50:20.477 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:50:20.552 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-07-14T17:50:20.554 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-07-14T17:50:20.555 DEBUG:teuthology.orchestra.run.smithi019:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-07-14T17:50:20.556 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:21.564 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:22.573 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:23.579 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:24.587 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:25.186 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:50:30.193 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:50:30.206 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:31.213 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:32.220 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:33.228 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:34.236 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:35.244 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:35.308 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T17:50:35.656 INFO:teuthology.orchestra.console:Power on for smithi031 completed
2023-07-14T17:50:35.757 INFO:teuthology.provision.fog.smithi031:Waiting for deploy to finish
2023-07-14T17:50:35.758 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-07-14T17:50:40.766 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:40.831 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-14T17:50:45.839 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:50:50.845 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:50.913 ERROR:teuthology.orchestra.connection:Error authenticating with smithi031.front.sepia.ceph.com: Authentication failed.
2023-07-14T17:50:50.914 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-07-14T17:50:50.914 DEBUG:teuthology.orchestra.run.smithi031:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-07-14T17:50:50.915 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:50.949 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:51.928 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:51.958 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:52.939 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:52.968 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:53.948 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:53.977 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:54.957 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:54.987 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:55.966 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:55.997 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:56.975 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:57.010 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:57.985 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:58.020 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:58.997 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:50:59.031 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:00.008 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:00.041 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:01.018 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:01.052 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:02.028 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:02.062 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:03.038 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:03.071 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:04.049 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:04.081 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:05.059 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:05.090 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:05.918 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:51:10.926 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:10.926 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:11.932 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:11.932 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:12.938 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:12.938 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:13.945 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:13.945 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:14.952 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:14.952 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:15.960 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:15.960 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:16.966 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:17.322 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:17.973 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:18.027 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:18.981 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:19.035 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:19.988 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:20.996 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:22.005 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:23.014 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:24.022 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:25.032 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:25.998 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:51:31.005 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:32.012 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:33.021 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:34.028 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:35.037 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:36.044 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:37.051 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:38.058 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:39.065 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:40.071 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:41.079 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:42.085 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:43.092 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:44.100 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:45.107 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:51:46.069 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:51:47.498 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-14T17:52:02.563 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:52:17.622 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:52:37.688 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:52:52.768 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:53:07.840 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:53:15.376 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104)
2023-07-14T17:53:15.378 ERROR:teuthology.provision.fog:[Errno 104] Connection reset by peer on smithi019
2023-07-14T17:53:24.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-07-14T17:53:24.380 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-14T17:53:29.385 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:53:34.713 DEBUG:teuthology.orchestra.run.smithi019:> true
2023-07-14T17:53:35.070 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-07-14T17:53:35.070 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-07-14T17:53:35.071 DEBUG:teuthology.orchestra.run.smithi019:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-07-14T17:53:35.089 INFO:teuthology.provision.fog.smithi019:Node is ready
2023-07-14T17:53:35.090 DEBUG:teuthology.orchestra.run.smithi019:> hostname
2023-07-14T17:53:35.148 INFO:teuthology.orchestra.run.smithi019.stdout:smithi019.front.sepia.ceph.com
2023-07-14T17:53:35.149 DEBUG:teuthology.orchestra.run.smithi019:> grep smithi019.front.sepia.ceph.com /etc/hosts
2023-07-14T17:53:35.207 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19 smithi019.front.sepia.ceph.com smithi019
2023-07-14T17:53:35.208 DEBUG:teuthology.orchestra.run.smithi019:> sudo hostname smithi019
2023-07-14T17:53:35.335 DEBUG:teuthology.orchestra.run.smithi019:> sudo sed -i -e 's/smithi019.front.sepia.ceph.com/smithi019/g' /etc/hosts
2023-07-14T17:53:35.365 DEBUG:teuthology.orchestra.run.smithi019:> sudo sed -i -e 's/172.21.15.19/172.21.15.19/g' /etc/hosts
2023-07-14T17:53:35.435 DEBUG:teuthology.orchestra.run.smithi019:> sudo sed -i -e 's/smithi019.front.sepia.ceph.com/smithi019/g' /etc/hostname
2023-07-14T17:53:35.503 DEBUG:teuthology.orchestra.run.smithi019:> sudo hostname smithi019
2023-07-14T17:53:35.571 DEBUG:teuthology.orchestra.run.smithi019:> cat /etc/os-release
2023-07-14T17:53:35.628 INFO:teuthology.orchestra.run.smithi019.stdout:NAME="Red Hat Enterprise Linux"
2023-07-14T17:53:35.629 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION="8.6 (Ootpa)"
2023-07-14T17:53:35.629 INFO:teuthology.orchestra.run.smithi019.stdout:ID="rhel"
2023-07-14T17:53:35.629 INFO:teuthology.orchestra.run.smithi019.stdout:ID_LIKE="fedora"
2023-07-14T17:53:35.629 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION_ID="8.6"
2023-07-14T17:53:35.629 INFO:teuthology.orchestra.run.smithi019.stdout:PLATFORM_ID="platform:el8"
2023-07-14T17:53:35.630 INFO:teuthology.orchestra.run.smithi019.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-07-14T17:53:35.630 INFO:teuthology.orchestra.run.smithi019.stdout:ANSI_COLOR="0;31"
2023-07-14T17:53:35.630 INFO:teuthology.orchestra.run.smithi019.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-07-14T17:53:35.630 INFO:teuthology.orchestra.run.smithi019.stdout:HOME_URL="https://www.redhat.com/"
2023-07-14T17:53:35.630 INFO:teuthology.orchestra.run.smithi019.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-07-14T17:53:35.630 INFO:teuthology.orchestra.run.smithi019.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-07-14T17:53:35.630 INFO:teuthology.orchestra.run.smithi019.stdout:
2023-07-14T17:53:35.631 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-07-14T17:53:35.631 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-07-14T17:53:35.631 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-07-14T17:53:35.631 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-07-14T17:53:35.632 INFO:teuthology.provision.fog.smithi019:Deploy complete!
2023-07-14T17:53:35.633 DEBUG:teuthology.parallel:result is None
2023-07-14T17:53:35.856 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104)
2023-07-14T17:53:35.857 ERROR:teuthology.provision.fog:[Errno 104] Connection reset by peer on smithi031
2023-07-14T17:53:44.865 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com'
2023-07-14T17:53:44.866 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-14T17:53:49.870 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:53:55.203 DEBUG:teuthology.orchestra.run.smithi031:> true
2023-07-14T17:53:55.508 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com'
2023-07-14T17:53:55.509 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-07-14T17:53:55.509 DEBUG:teuthology.orchestra.run.smithi031:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-07-14T17:53:55.526 INFO:teuthology.provision.fog.smithi031:Node is ready
2023-07-14T17:53:55.527 DEBUG:teuthology.orchestra.run.smithi031:> hostname
2023-07-14T17:53:55.584 INFO:teuthology.orchestra.run.smithi031.stdout:smithi031.front.sepia.ceph.com
2023-07-14T17:53:55.585 DEBUG:teuthology.orchestra.run.smithi031:> grep smithi031.front.sepia.ceph.com /etc/hosts
2023-07-14T17:53:55.645 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31 smithi031.front.sepia.ceph.com smithi031
2023-07-14T17:53:55.645 DEBUG:teuthology.orchestra.run.smithi031:> sudo hostname smithi031
2023-07-14T17:53:55.778 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i -e 's/smithi031.front.sepia.ceph.com/smithi031/g' /etc/hosts
2023-07-14T17:53:55.807 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i -e 's/172.21.15.31/172.21.15.31/g' /etc/hosts
2023-07-14T17:53:55.877 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i -e 's/smithi031.front.sepia.ceph.com/smithi031/g' /etc/hostname
2023-07-14T17:53:55.945 DEBUG:teuthology.orchestra.run.smithi031:> sudo hostname smithi031
2023-07-14T17:53:56.012 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release
2023-07-14T17:53:56.070 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="Red Hat Enterprise Linux"
2023-07-14T17:53:56.071 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="8.6 (Ootpa)"
2023-07-14T17:53:56.071 INFO:teuthology.orchestra.run.smithi031.stdout:ID="rhel"
2023-07-14T17:53:56.071 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="fedora"
2023-07-14T17:53:56.071 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="8.6"
2023-07-14T17:53:56.071 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el8"
2023-07-14T17:53:56.072 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-07-14T17:53:56.072 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31"
2023-07-14T17:53:56.072 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-07-14T17:53:56.072 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://www.redhat.com/"
2023-07-14T17:53:56.072 INFO:teuthology.orchestra.run.smithi031.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-07-14T17:53:56.072 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-07-14T17:53:56.072 INFO:teuthology.orchestra.run.smithi031.stdout:
2023-07-14T17:53:56.073 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-07-14T17:53:56.073 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-07-14T17:53:56.073 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-07-14T17:53:56.073 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-07-14T17:53:56.073 INFO:teuthology.provision.fog.smithi031:Deploy complete!
2023-07-14T17:53:56.074 DEBUG:teuthology.parallel:result is None
2023-07-14T17:54:09.947 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:54:30.012 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:54:50.081 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:55:10.154 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:55:30.239 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:55:46.932 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:56:06.998 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:56:27.068 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:56:47.136 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:57:07.202 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:57:27.267 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:57:42.336 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:57:57.405 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:58:12.478 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:58:32.558 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:58:52.635 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:59:12.712 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:59:27.977 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T17:59:48.050 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:00:08.128 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:00:28.192 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:00:48.259 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:01:08.310 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:01:23.376 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:01:38.439 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:01:58.500 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:02:17.571 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:02:32.640 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:02:52.707 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:03:07.772 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:03:22.864 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:03:42.935 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:04:03.004 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:04:18.063 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:04:38.119 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:04:58.177 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:05:13.240 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:05:28.435 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:05:43.510 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:05:58.584 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:06:13.652 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:06:32.974 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:06:53.088 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:07:13.199 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:07:28.292 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:07:43.390 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:08:03.465 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:08:18.583 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:08:38.657 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:08:58.713 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:09:18.772 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:09:38.830 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:09:58.892 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:10:13.947 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:10:31.415 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:10:46.501 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:11:06.584 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:11:21.662 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:11:41.726 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:12:01.789 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:12:21.878 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:12:41.958 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:12:57.026 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:13:14.410 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:13:34.476 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:13:54.543 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:14:14.639 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:14:29.731 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:14:49.820 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:15:09.893 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:15:29.965 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:15:48.079 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:16:03.167 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:16:23.263 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:16:38.340 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:16:55.139 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:17:10.230 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:17:30.341 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:17:45.438 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:18:00.527 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:18:15.623 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:18:35.613 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:18:55.702 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:19:15.775 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:19:35.873 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:19:55.933 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:20:16.011 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:20:31.086 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:20:46.156 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:21:06.216 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:21:26.283 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:21:46.341 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:22:06.403 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:22:26.468 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:22:43.567 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:23:03.646 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:23:23.711 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:23:43.788 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:24:03.851 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:24:18.916 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:24:36.901 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:24:56.994 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:25:17.085 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:25:37.173 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:25:57.246 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:26:17.339 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:26:32.414 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:26:52.500 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:27:07.568 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:27:27.651 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:27:47.720 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:28:07.060 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:28:27.155 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:28:47.242 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:29:07.321 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:29:27.407 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:29:47.494 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:30:07.573 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:30:27.672 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:30:42.766 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:31:02.844 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:31:17.916 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:31:37.997 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:31:58.072 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:32:16.275 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:32:31.390 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:32:48.669 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:33:08.746 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:33:28.890 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:33:48.971 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:34:09.062 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:34:29.131 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:34:49.195 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:35:04.261 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:35:19.317 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:35:39.373 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:35:54.427 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:36:09.480 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:36:24.544 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:36:39.606 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:36:54.672 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:37:14.739 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:37:29.801 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:37:49.880 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:38:09.946 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:38:25.069 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:38:45.140 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:39:05.202 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:39:25.267 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:39:45.330 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:40:05.393 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:40:25.464 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:40:40.530 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:40:55.675 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:41:15.739 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:41:30.805 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:41:45.877 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:42:05.949 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:42:21.002 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:42:41.070 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:43:01.134 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:43:21.192 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:43:36.245 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:43:56.303 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:44:16.358 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:44:36.413 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:44:51.475 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:45:11.534 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:45:31.593 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:45:51.654 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:46:11.730 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:46:31.803 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:46:46.904 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:47:06.229 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:47:26.315 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:47:41.411 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:47:57.835 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:48:14.035 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:48:29.143 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:48:49.214 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:49:09.292 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:49:24.390 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:49:44.462 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:49:59.527 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:50:19.614 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:50:35.602 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:50:55.680 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:51:10.765 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:51:25.863 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:51:45.947 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:52:04.027 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:52:24.114 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:52:44.215 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:52:59.804 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:53:14.927 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:53:34.854 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:53:54.971 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:54:15.079 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:54:30.184 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:54:50.272 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:55:10.349 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:55:30.413 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:55:49.073 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:56:04.147 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:56:24.222 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:56:44.311 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:56:59.396 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:57:14.468 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:57:34.545 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:57:54.615 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:58:14.334 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:58:34.415 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:58:54.035 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:59:14.118 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:59:29.323 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T18:59:49.303 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T19:00:09.370 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T19:00:29.436 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T19:00:44.489 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T19:00:59.542 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T19:01:14.598 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T19:01:29.657 DEBUG:teuthology.provision.fog.smithi202:FOG request: GET /task/active None
2023-07-14T19:01:34.715 DEBUG:teuthology.provision.fog.smithi202:Canceling deploy task with ID 384757
2023-07-14T19:01:34.716 DEBUG:teuthology.provision.fog.smithi202:FOG request: DELETE /task/cancel {"id": 384757}
2023-07-14T19:01:34.791 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/dispatcher/supervisor.py", line 227, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/provision/fog.py", line 86, in create
    self.wait_for_deploy_task(task_id)
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/provision/fog.py", line 259, in wait_for_deploy_task
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/contextutil.py", line 134, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: reached maximum tries (241) after waiting for 3600 seconds
2023-07-14T19:01:34.792 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-14T19:01:34.793 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-14T19:01:38.823 INFO:teuthology.nuke:Checking targets against current locks
2023-07-14T19:01:48.846 DEBUG:teuthology.nuke:shortname: smithi019
2023-07-14T19:01:48.847 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-14T19:01:48.872 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi019.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338006', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 17:46:54.264105', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}
2023-07-14T19:01:48.875 DEBUG:teuthology.nuke:shortname: smithi031
2023-07-14T19:01:48.876 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-14T19:01:48.896 DEBUG:teuthology.nuke:shortname: smithi202
2023-07-14T19:01:48.897 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-14T19:01:48.901 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338006', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 17:46:54.260998', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='}
2023-07-14T19:01:48.929 INFO:teuthology.orchestra.console:Power off smithi019
2023-07-14T19:01:48.930 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-14T19:01:48.944 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi202.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338006', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 17:46:54.262598', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD2mw+DA1k8VmYGdQOB3AvKB1kHrCoqHf5vkDJy7kOX6Pt4wEQwecHn76zTBWlUKsGSmTWDSeE6EzsGiZbFGmtDce8U9gr2ZG/yi8q/qlE3giw2SSqYnp6HRyTqRYzshY2FPHP96F9v5ZVME46myhQw++MsPG4xypKWHSuYTFbcUt3an+hToPSTevtSRyQKzac0Wud++HVmMCvUb+GHk+4FXn6S5LrNzIdsH9oQLZ+mOZkJcdxkawF2PAzMSIOILj+Sdb6dy2KjDshYp/RN6Sqvh1MjITL1D03h1ophAJfm6CBxAV3vEdLs1Y0dV9ai4wHZoKb+Df88hW0syXiK1v3pm/SisDKAQ7k3Fc/5gYMI3CpGbjxYvjmkWsZ/xDvVTy6brNdMutGUrAN0TH887x4rNn3/Qp7c2uzT5XLDdS+Wl1ot83BPd0AQkxC8cdX4Iu1Isnhj0gWh2NYF9zeTRlBQ16mwrBw4mxMXOo7O6xtGRoM7i68MORZFzH8+23lZ4Q0='}
2023-07-14T19:01:58.957 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T19:01:58.991 INFO:teuthology.orchestra.console:Power off smithi031
2023-07-14T19:01:58.992 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-14T19:02:04.006 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T19:02:04.037 INFO:teuthology.orchestra.console:Power off smithi202
2023-07-14T19:02:04.037 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi202.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-14T19:02:04.057 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi202.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T19:02:08.008 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T19:02:08.031 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T19:02:08.080 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi202.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T19:02:08.219 INFO:teuthology.orchestra.console:Power off for smithi019 completed
2023-07-14T19:02:08.242 INFO:teuthology.orchestra.console:Power off for smithi031 completed
2023-07-14T19:02:13.330 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi202.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-14T19:02:23.373 INFO:teuthology.orchestra.console:Power off for smithi202 completed
2023-07-14T19:02:30.243 INFO:teuthology.lock.ops:unlocked: smithi019.front.sepia.ceph.com
2023-07-14T19:02:35.278 INFO:teuthology.lock.ops:unlocked: smithi031.front.sepia.ceph.com
2023-07-14T19:02:35.312 INFO:teuthology.lock.ops:unlocked: smithi202.front.sepia.ceph.com
2023-07-14T19:02:40.715 ERROR:teuthology.dispatcher.supervisor:Reimage failed 10 times ... marking machine down
2023-07-14T19:02:40.716 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/dispatcher/supervisor.py", line 57, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/dispatcher/supervisor.py", line 227, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/provision/fog.py", line 86, in create
    self.wait_for_deploy_task(task_id)
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/provision/fog.py", line 259, in wait_for_deploy_task
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/contextutil.py", line 134, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: reached maximum tries (241) after waiting for 3600 seconds