2023-03-09T14:18:09.485 INFO:root:teuthology version: 0.0.1.dev51+ga2dfb05
2023-03-09T14:18:09.674 INFO:teuthology.lock.ops:Start node 'smithi139.front.sepia.ceph.com' reimaging
2023-03-09T14:18:09.676 INFO:teuthology.lock.ops:Updating [smithi139.front.sepia.ceph.com]: reset os type and version on server
2023-03-09T14:18:09.676 INFO:teuthology.lock.ops:Updating smithi139.front.sepia.ceph.com on lock server
2023-03-09T14:18:09.709 INFO:teuthology.lock.ops:Node 'smithi139.front.sepia.ceph.com' reimaging is complete
2023-03-09T14:18:09.710 INFO:teuthology.lock.ops:Start node 'smithi140.front.sepia.ceph.com' reimaging
2023-03-09T14:18:09.710 INFO:teuthology.lock.ops:Updating [smithi140.front.sepia.ceph.com]: reset os type and version on server
2023-03-09T14:18:09.710 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server
2023-03-09T14:18:09.741 INFO:teuthology.lock.ops:Node 'smithi140.front.sepia.ceph.com' reimaging is complete
2023-03-09T14:18:09.879 INFO:teuthology.provision.fog.smithi139:Scheduling deploy of rhel 8.6
2023-03-09T14:18:09.899 INFO:teuthology.provision.fog.smithi140:Scheduling deploy of rhel 8.6
2023-03-09T14:18:10.150 INFO:teuthology.orchestra.console:Power off smithi139
2023-03-09T14:18:10.168 INFO:teuthology.orchestra.console:Power off smithi140
2023-03-09T14:18:18.537 INFO:teuthology.orchestra.console:Power off for smithi140 completed
2023-03-09T14:18:18.638 INFO:teuthology.orchestra.console:Power on smithi140
2023-03-09T14:18:22.642 INFO:teuthology.orchestra.console:Power off for smithi139 completed
2023-03-09T14:18:22.744 INFO:teuthology.orchestra.console:Power on smithi139
2023-03-09T14:18:27.001 INFO:teuthology.orchestra.console:Power on for smithi139 completed
2023-03-09T14:18:27.009 INFO:teuthology.orchestra.console:Power on for smithi140 completed
2023-03-09T14:18:27.102 INFO:teuthology.provision.fog.smithi139:Waiting for deploy to finish
2023-03-09T14:18:27.110 INFO:teuthology.provision.fog.smithi140:Waiting for deploy to finish
2023-03-09T14:21:31.525 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.139
2023-03-09T14:21:31.527 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:21:40.581 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.139
2023-03-09T14:21:40.613 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:21:49.637 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.139
2023-03-09T14:21:49.673 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:21:58.725 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:22:07.781 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:22:16.837 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:22:55.673 WARNING:teuthology.provision.fog:timed out
2023-03-09T14:23:01.952 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-09T14:23:02.481 INFO:teuthology.provision.fog.smithi139:Node is ready
2023-03-09T14:23:02.499 INFO:teuthology.orchestra.run.smithi139.stdout:smithi139.front.sepia.ceph.com
2023-03-09T14:23:02.558 INFO:teuthology.orchestra.run.smithi139.stdout:172.21.15.139 smithi139.front.sepia.ceph.com smithi139
2023-03-09T14:23:02.908 INFO:teuthology.provision.fog.smithi139:Deploy complete!
2023-03-09T14:23:22.861 WARNING:teuthology.provision.fog:timed out
2023-03-09T14:23:31.947 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:23:41.033 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:23:50.117 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:23:59.205 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:24:08.261 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:24:17.317 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:24:26.378 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:24:35.461 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:24:44.521 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:24:53.609 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:25:02.697 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:25:11.781 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:25:20.837 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:25:29.897 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:25:38.981 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:25:48.041 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:25:57.129 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:26:06.213 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:26:15.301 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:26:24.357 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:26:33.413 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:26:42.473 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:26:51.557 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:27:00.613 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:27:09.677 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:27:18.757 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:27:27.813 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:27:36.901 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:27:45.961 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:27:55.045 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:28:04.101 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:28:13.157 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:28:22.217 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:28:31.301 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:28:40.357 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:28:49.413 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:28:58.469 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:29:07.525 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:29:16.585 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:29:25.669 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:29:34.725 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:29:43.781 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:29:52.837 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:30:01.893 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:30:10.949 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:30:20.041 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:30:29.125 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:30:38.181 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:30:47.237 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:30:56.297 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:31:05.385 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:31:14.473 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:31:23.557 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:31:32.613 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:31:41.669 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:31:50.729 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:31:59.813 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:32:08.873 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:32:17.957 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:32:27.017 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:32:36.101 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:32:45.157 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:32:54.217 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:33:03.301 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:33:12.357 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:33:21.445 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:33:30.505 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:33:39.593 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:33:48.681 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:33:57.769 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:34:06.853 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:34:15.913 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:34:25.001 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:34:34.089 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:34:43.173 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:34:52.229 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:35:01.289 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:35:10.373 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:35:19.433 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:35:28.521 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:35:37.609 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:35:46.693 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:35:55.749 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:36:04.805 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:36:13.893 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:36:22.949 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:36:32.005 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:36:41.065 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:36:50.153 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:36:59.241 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:37:08.329 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:37:17.413 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:37:26.469 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-09T14:37:26.471 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/dispatcher/supervisor.py", line 209, in reimage
    reimaged = reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/lock/ops.py", line 324, in reimage_machines
    log.info("Node '%s' reimaging is complete", machine)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/provision/__init__.py", line 39, in reimage
    return obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/provision/fog.py", line 90, in create
    self._wait_for_ready()
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/provision/fog.py", line 271, in _wait_for_ready
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/contextutil.py", line 135, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds
2023-03-09T14:37:26.557 INFO:teuthology.nuke:Checking targets against current locks
2023-03-09T14:37:26.613 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-09T14:37:26.632 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-09T14:37:26.695 INFO:teuthology.orchestra.console:Power off smithi139
2023-03-09T14:37:26.711 INFO:teuthology.orchestra.console:Power off smithi140
2023-03-09T14:37:35.085 INFO:teuthology.orchestra.console:Power off for smithi139 completed
2023-03-09T14:37:35.091 INFO:teuthology.orchestra.console:Power off for smithi140 completed
2023-03-09T14:37:35.245 INFO:teuthology.lock.ops:unlocked: smithi139.front.sepia.ceph.com
2023-03-09T14:37:35.261 INFO:teuthology.lock.ops:unlocked: smithi140.front.sepia.ceph.com
2023-03-09T14:37:35.503 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/dispatcher/__init__.py", line 63, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/dispatcher/supervisor.py", line 52, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/dispatcher/supervisor.py", line 209, in reimage
    reimaged = reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/lock/ops.py", line 324, in reimage_machines
    log.info("Node '%s' reimaging is complete", machine)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/provision/__init__.py", line 39, in reimage
    return obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/provision/fog.py", line 90, in create
    self._wait_for_ready()
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/provision/fog.py", line 271, in _wait_for_ready
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/contextutil.py", line 135, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: reached maximum tries (100) after waiting for 600 seconds